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By 
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Chairman:  Victor  A.  Thompson 
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This  work  analyzes  the  role  that  evaluation  plays  in  a group 
of  mental  health  organizations.  It  examines  the  hypothesis  that 
organizations  are  incapable  of  critically  evaluating  their  own  activities 
due  to  the  priority  given  to  requirements  for  survival. 

Formal,  scientific  evaluation  is  distinguished  from  less  formal 
and  more  subjective  types  of  evaluation.  Three  dimensions  of 
evaluation  are  developed  in  order  to  distinguish  "critical"  evaluations 
from  "token"  evaluations:  (1)  the  scope  of  the  activities  of  the 
organization  covered  by  the  evaluation;  (2)  the  focus  of  the  evaluation-- 
whether  it  is  oriented  towards  measuring  intermediate  or  end  goals; 

(3)  the  substance  of  the  evaluation --how  much  change  actually  results 
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from  the  introduction  of  the  evaluation. 


The  research  problem  is  seen  as  being  related  to  four  important 
areas  of  organizational  and  public  administration  theory:  (1)  the  extent 
to  which  organizations  are  survival  or  goal-oriented;  (2)  the  resistance 
of  the  "natural"  system  of  the  organization  to  the  encroachment  of 
formal  aspects  of  the  organization;  (3)  the  importance  of  the  technology 
of  organizations;  and  (4)  the  issues  of  accountability  and  responsibility 
of  public  organizations. 

The  sample  of  organizations  studied  includes  fourteen  mental 
health  centers  and  clinics  in  the  State  of  Florida.  The  methods  used 
include  five  months  of  participant  observation  in  one  mental  health 
center,  and  interviews  and  questionnaires  administered  to  clinicians, 
evaluators,  and  directors  of  the  above  organizations. 

One  finding  is  that  evaluation  has  little  impact  on  those  organi- 
zations in  which  it  is  introduced.  However,  there  is  little  evidence 
of  overt  opposition  to  evaluation.  It  is  introduced  in  such  a non- 
threatening way  that  it  is  not  expected  to  play  a significant  role  in 
decision  making. 

Organizations  that  voluntarily  undertake  formal  evaluation 
expect  it  to  contribute  to  the  survival  of  the  organization  by  providing 
a means  of  justifying  the  public  funds  spent  to  support  them,  by 
helping  to  secure  grants,  and  by  giving  them  greater  influence  over 
possible  future  evaluations  by  outsiders. 
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Four  major  factors  are  seen  as  important  in  determining 
whether  an  evaluator  is  able  to  achieve  responsiveness  to  the  values 
he  wishes  to  emphasize:  (1)  the  kinds  of  values  held  by  the  evaluators 
and  the  persons  being  evaluated;  (2)  the  extent  to  which  the  evaluators 
are  motivated  to  achieve  responsiveness  to  their  values;  (3)  the  types 
of  authority  exercised  by  the  evaluator  and  the  reaction  of  the  persons 
being  evaluated  to  these  types;  and  (4)  the  extent  to  which  evaluators 
possess  information  to  determine  whether  or  not  their  evaluations 
had  their  intended  effect. 

Generally,  there  is  confirmation  of  the  thesis  that  critical 
evaluation  is  rare  in  organizations  because  of  the  tendency  of 
activities  most  closely  associated  with  organizational  survival 
to  displace  other  types  of  activities.  Certain  factors  which  increase 
the  likelihood  of  critical  evaluation  in  organizations  are  noted. 
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CHAPTER  I 


INTRODUCTION 
The  Problem 

The  original  inspiration  for  this  dissertation  was  a simple 
yet  intriguing  question  raised  by  Aaron  Wildavsky:  "Can  organi- 
zations evaluate  themselves?"^  Wildavsky  argues  that  organizations 
and  the  people  composing  them  generate  such  strong  commitments 
to  programs,  goals,  clientele,  and  to  the  adaptive  needs  of  survival 
in  general,  that  self-evaluation  is  impossible.  However,  since 
evaluation  would  cost  more  than  it  is  worth  unless  it  leads  to  sub- 
stantive change,  evaluators  within  organizations  need  the  power, 
stability,  and  additional  resources  that  can  force  the  other  people 
in  the  organization  to  listen  to  them  and  accept  the  recommendations 
based  on  their  evaluations.  Yet,  the  very  pursuit  of  these  kinds  of 
resources  is  likely  to  displace  the  evaluators  from  their  original 
goals  and  lead  them  to  new  commitments  which  are  very  similar 

*Aaron  Wildavsky,  "The  Self-Evaluating  Organization,  " 

Public  Administration  Review,  Vol.  32  (November/December,  1972), 
pp.  509-520. 

^ "Evaluation"  will  be  defined  below. 
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to  those  held  by  others  in  the  organization.  The  implication  of 
Wildavsky's  analysis  is  that  evaluation  is  likely  to  result  in  little 
substantive  change,  and  the  evaluators  are  likely  to  be  either 
frustrated  or  coopted. 

3 

This  study  examines  the  role  that  evaluation  has  played 
in  a group  of  organizations  and  the  question  raised  by  Wildavsky. 

It  is  an  organizational  study  which  explores  the  relationship  between 
evaluative  behavior  in  organizations  and  organizational  elements 
such  as  formal  authority,  the  informal  system,  etc.  ^ The  examina- 
tion of  the  role  of  evaluation  in  organizations  can  be  broken  down 
into  sub -areas  of  research  questions  which  are  explored  in  this 
dissertation,  including  the  following: 

1.  What  impact  has  evaluation  had  on  the  group  of 
organizations  under  study?  What  part  does  it  play  in  the 
decision  making  that  takes  place  in  the  organizations? 

2.  What  are  the  attitudes  towards  evaluation  of  the 
individuals  composing  the  organizations  being  studied? 

3.  Wliat  are  the  forces  which  lead  organizations  (in 
this  study)  to  invest  in  undertaking  evaluations?  Why  do 
certain  organizations  invest  more  heavily  in  evaluation 
than  others  ? 

4.  What  are  the  factors  involved  which  are  likely  to 
make  evaluators  successful  in  achieving  the  goals  of  their 
evaluations  ? 


ShRole"  is  being  used  broadly  here  to  refer  to  both  the  actual 
and  anticipated  consequences  of  evaluation.  These  consequences 
include  both  manifest  and  latent  functions,  and  functional  and 
dysfunctional  consequences. 

^See  below  for  a further  discussion  of  these  and  other  elements 
of  organizational  theory. 
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Community  Mental  Health  Centers 

The  research  for  this  dissertation  was  carried  out  on 

thirteen  community  mental  health  centers  and  one  clinic  in  the 

State  of  Florida.  ^ Community  mental  health  centers  were  established 

in  the  1960's  for  several  reasons  including  the  goals  of  providing 

psychiatric  services  to  those  who  could  not  afford  private  therapists, 

of  putting  more  emphasis  upon  prevention  of  mental  illness,  and  of 

reducing  the  large  state  mental  hospital  loads.  Organizations 

designated  as  mental  health  centers  are  distinguished  by  the  fact 

that  they  have  received  support  from  the  National  Institute  of  Mental 

Health  (NIMH)  in  return  for  which  they  offer  as  a minimum  the 

following  services:  (1)  outpatient  care,  (2)  inpatient  care, 

(3)  emergency  care,  (4)  partial  hospitalization  care,  (5)  consultation 

7 

and  education  programs.  The  area  served  by  each  mental  health 
center  is  called  its  "catchment  area.  " A district  board  composed  of 


^The  thirteen  community  mental  health  centers  included  the 
entire  population  of  centers  which  had  already  received  funded 
staffing  grants  by  the  beginning  of  this  study.  The  reason  for  the 
inclusion  of  the  one  clinic  was  that  it  was  highly  advanced  with 
respect  to  evaluation. 

0 

See  Joint  Commission  on  Mental  Illness  and  Health,  Action 
for  Mental  Health  (New  York:  Basic  Books,  1961). 

n 

See  Allan  Beigel  and  Alan  Levenson,  "The  Community  Mental 
Health  Center:  Strategies  and  Concepts,  " in  The  Community  Mental 
Health  Center:  Strategies  and  Programs,  ed.  Allan  Beigel  and  Alan 
Levenson  (New  York:  Basic  Books,  1972),  pp.  3-16. 
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members  from  the  catchment  area  oversees  the  operations  of  each 
mental  health  center.  The  mental  health  centers  in  Florida  also 
receive  financial  support  from  State  Division  of  Mental  Health. 

Community  mental  health  centers  offer  a particularly  fruit- 
ful opportunity  to  shady  the  role  of  evaluation  in  organizations  and 
the  above  research  questions  because  there  is  substantial  evidence 
that  they  are  among  the  most  advanced  of  service  organizations  in 
implementing  formal  evaluation.  The  National  Institute  of  Mental 
Health  and  community  mental  health  centers  have  devoted  more 

O 

resources  to  evaluation  than  any  other  health  agencies.  In  an 
interview  concerning  evaluation  in  the  Department  of  Health,  Edu- 
cation, and  Welfare  (HEW),  the  then  Secretary  of  HEW,  Elliot 

Richardson,  cited  community  mental  health  centers  as  being  the 

g 

best  example  of  where  evaluation  results  had  been  put  to  use. 

A study  carried  out  by  NIMH  found  that  program  evaluation  had 
become  the  single  most  important  concern  of  mental  health  centers. 

In  particular,  the  mental  health  centers  and  clinics  in  the 
State  of  Florida  appeared  to  be  in  the  forefront  of  the  movement 

^See  Garth  Buchanan  and  Joseph  S.  Wholey,  "Federal  Level 
Evaluation,  " Evaluation,  Vol.  1,  No.  1 (1972),  p.  19. 

^Susan  Salasin,  "Conversational  Contact  with  Elliot  Richardson,  " 
Evaluation,  Vol.  1,  No.  1 (1972),  p.  16. 

* ^Howard  Davis,  "A  Solution  for  a Crisis?  " Evaluation,  Vol.  1, 
No.  1 (1972),  p.  3. 
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to  experiment  with  evaluation.  Nearly  all  of  these  organizations 

had  introduced  or  were  planning  to  develop  some  form  of  formalized 

evaluation. 11  In  short,  it  seemed  that  the  role  of  evaluation  was 

likely  to  be  at  least  as  highly  articulated  in  this  group  of  mental 

health  organizations  as  in  nearly  any  other  type  of  public  organization. 

Furthermore,  because  they  were  advanced  with  respect  to  evaluation, 

they  would  provide  a good  opportunity  to  examine  the  thesis  that 

organizations  are  incapable  of  evaluating  themselves.  Glaser  and 

Strauss  have  argued  for  what  they  call  "theoretical  sampling"  which 

includes  the  deliberate  search  for  cases  which  seem  to  be  contrary 

1 2 

to  the  theory  or  hypothesis  being  tested. 

The  Literature 

Not  all  social  scientists  are  as  pessimistic  as  Wildavsky 
concerning  the  utility  of  evaluation.  Moynihan  argues  that  the  social 
sciences  are  at  their  worst  in  offering  theories  of  how  to  bring  about 
massive  behavioral  change  and  that  the  "correct"  role  of  the  social 
sciences  lies  not  in  the  formulation  of  social  policy  but  in  the 

11  At  a meeting  attended  by  the  author,  NIMH  officials  indicated 
that  the  mental  health  centers  included  in  the  group  being  studied 
here  were  relatively  advanced  with  respect  to  evaluation  in  comparison 
with  most  other  mental  health  centers. 

1 2 

Barney  Glaser  and  Anselm  Strauss,  The  Discovery  of 
Grounded  Theory:  Strategies  for  Qualitative  Research  (Chicago: 

Aldine  Publishing  Company,  1967),  p.  56. 
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measurement  of  its  results.  Rivlin  states  that  in  the  last  decade 

much  progress  has  been  made  concerning  our  knowledge  of  the 

prevalance  and  incidence  of  social  problems  and  that  we  now  need 

to  concentrate  more  on  how  to  produce  effective  programs  and  measure 

14 

the  costs  and  benefits  of  these  programs.  Allen  Schick  argues  that 

evaluation  is  likely  to  be  more  productive  than  past  innovations  such 

as  program  budgeting  and  policy  analysis  for  the  following  reasons: 

(1)  the  focus  of  evaluation  is  more  limited  and  hence  manageable 

than  that  of  policy  analysis;  and  (2)  evaluation  is  retrospective  and 

thus  more  compatible  with  the  budgetary  process  than  program 

budgeting  and  policy  analysis.  Donald  Campbell  also  states  that 

the  role  of  the  social  scientist  should  be  to  create  a public  demand 

for  hard-headed  evaluations  and  to  be  committed  to  the  solving  of 

1 6 

problems  rather  than  to  the  particular  means  of  solving  them. 

Actually,  even  those  who  are  generally  enthusiastic  about  the 
concept  of  evaluation  are  pessimistic  about  the  possibility  of 


^ Daniel  P.  Moynihan,  Maximum  Feasible  Misunderstanding 
(New  York:  Free  Press,  1969),  pp.  191-196. 

14 

Alice  M.  Rivlin,  Systematic  Thinking  for  Social  Action 
(Washington:  Brookings  Institution,  1971),  pp.  7,  60. 

^Allen  Schick,  "From  Analysis  to  Evaluation,  " The  Annals 
of  the  American  Academy  of  Political  and  Social  Science,  Vol.  385 
(September,  1969),  pp.  61-70. 

1 fi 

Donald  T.  Campbell,  "Reforms  as  Experiments,  " American 
Psychologist,  Vol.  124  (April,  1969),  pp.  409-429. 
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organizations  evaluating  themselves.  Moynihan  calls  for  an  office 

of  legislative  evaluation  because  of  the  natural  tendency  to  oversell 

one's  programs  and  because  of  the  need  to  ensure  the  honesty  of 
1 7 

the  evaluators.  Schick  notes  that  in  order  for  evaluation  to  be  taken 

1 8 

seriously  it  must  be  linked  with  funding.  Rivlin  agrees  that  there 

will  be  a great  tendency  to  cheat  in  projects  that  are  being  evaluated 

19 

and  consequently  an  independent  evaluation  is  necessary. 

Campbell  points  out  several  plausible  but  questionable  ways 

in  which  administrators  may  use  data  to  demonstrate  the  success 
2 0 

of  their  reforms.  Edward  Suchman  has  even  created  a classifi- 
cation of  strategies  for  which  management  and  administrators  may 
use  evaluation:  (1)  eyewash:  using  evaluation  to  justify  weak  or  bad 
programs  by  selecting  out  good  aspects  for  evaluation,  (2)  whitewash: 
using  evaluations  to  cover  up  failures  by  avoiding  any  objective 
appraisal,  (3)  submarine:  using  evaluation  to  destroy  a program, 

(4)  postponement:  using  evaluation  to  delay  needed  action  by  pre- 

21 

tending  to  seek  or  research  other  factors.  Thus  most  experts 

17 

Moynihan,  Maximum  Feasible  Misunderstanding,  p.  200. 

1 8 

Schick,  "From  Analysis  to  Evaluation,"  p.  70. 

1 9 

Rivlin,  Systematic  Thinking  for  Social  Action,  p.  409. 

^Campbell,  "Reforms  as  Experiments,"  pp.  420-429. 

21 

Edward  A.  Suchman,  "Action  for  What?  A Critique  of 
Evaluative  Research,  " in  Evaluating  Action  Programs:  Readings  in 
Social  Action  and  Education,  ed.  Carol  H.  Weiss  (Boston:  Allyn 
and  Bacon,  1972),  pp.  52-84. 


8 


favor  an  "outsider"  approach  to  evaluation  because  they,  like 
Wildavsky,  have  strong  doubts  about  whether  or  not  public  organi- 

• 99 

zations  can  critically  evaluate  themselves. 

However,  there  is  little  evidence  to  support  the  proposition 
that  outsider  evaluation  is  more  effective  than  insider  evaluation.  22 
A recent  review  of  on-going  evaluation  in  the  federal  government 
concludes  as  follows: 

Despite  the  increased  evaluation  activity  that  we  will 
describe  in  the  survey,  the  present  evaluation  picture  is 
not  impressive  in  terms  of  identifiable  impact  on  policy 
decisions  and  program  operations.  24 

The  reasons  for  the  failure  of  contractual  or  outsider  evaluation 

are  numerous,  and  include  anxiety,  suspicion,  lack  of  cooperation 

and  sabotage  on  the  part  of  the  operating  personnel,  especially  when 

the  findings  reveal  anything  negative  concerning  the  operations  of 

the  programs.  22 


^^See,  e.  g.  , Edward  A.  Suchman,  Evaluative  Research: 
Principles  and  Practice  in  Public  Research  (New  York:  Russell 
Sage,  1967),  p.  18. 

23 

Even  Suchman,  generally  a proponent  of  outsider  evaluation, 
admits  that  insider  evaluations  are  more  likely  to  have  an  impact 
on  the  operations  of  the  organization.  See  Edward  Suchman,  Evaluative 
Research,  pp.  158,  163. 

24Garth  N.  Buchanan  and  Joseph  S.  Wholey,  "Federal  Level 
Evaluation,  " Evaluation,  Vol.  1,  No.  1 (1972),  p.  17. 

25 

For  a description  of  the  reaction  of  individuals  in  one  organi- 
zation to  a negative  evaluation,  see  James  A.  Davis,  "Great  Books 
and  Small  Groups:  An  Informal  History  of  a National  Survey,  " in 
Sociologists  at  Work,  ed.  Phillip  Hammond  (New  York:  Basic  Books, 
1964),  pp.  212-234. 
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From  an  organizational  perspective,  evaluation  must  become 
institutionalized  as  an  integral  component  of  the  organization's 
decision  making  system  which  a "one  shot"  outsider  evaluation 
simply  cannot  accomplish.  Thus,  three  experts  who  review  the 
problems  of  recent  evaluation  efforts  come  to  similar  conclusions -- 
that  emphasis  should  be  given  to  tying  evaluation  to  the  decision 
making  system  of  the  organization  and  that  more  "insider"  evaluation 
should  be  done: 

First,  resources  must  be  shifted  from  contract  work 
to  in-house  evaluation  design  efforts.  In  the  past,  federal 
agency  evaluation  offices  have  relied  solely  on  contractors 
for  design  work,  partly  because  the  cost  of  an  in-house 
technical  staff  was  considered  too  high.  But  it  is  our  view 
that  the  price  of  this  practice  in  unsatisfactory  evaluation  is 
higher,  really,  than  the  cost  of  having  the  design  function 
carried  out  by  in-house  staff.  2 6 

It  seems  that,  despite  reservations  of  nearly  everyone,  more 
insider  or  internal  organizational  evaluation  will  be  done  by  public 
organizations,  which  makes  the  problem  being  addressed  here 
timely:  "What  role  does  insider  or  self-evaluation  play  in  organiza- 
tions?"27 


2 fi 

°Garth  Buchanan,  Pamela  Horst,  and  John  Scanlon, 
"improving  Federal  Evaluation  Planning,"  Evaluation,  Vol.  1, 

No.  2 (1973),  p.  90. 

27 

However,  comparisons  will  be  made  with  external  forms 
of  evaluation  when  such  comparisons  aid  in  the  analysis  of  the  role 
of  internal  evaluation. 
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Concepts 

This  work  explores  the  relationship  between  two  basic  concepts: 
evaluation  and  organization.  Originally,  the  following  definition 
of  evaluation  was  developed:  "evaluation  is  the  formal  process  of 
measuring  the  attainment  of  goals,  however  defined,  through  the 

9 ft 

employment  of  the  tools  of  science.  The  author  had  hoped  to 

delimit  the  field  of  study  by  avoiding  less  formal  and  scientific 

forms  of  evaluation  such  as  the  "gut-feeling"  or  highly  subjective 

forms  of  judgments  which  take  place  in  nearly  all  forms  of  human 
29 

interaction.  However,  while  the  original  definition  of  evaluation 
was  retained,  it  became  apparent  that  the  relationship  between 
these  more  traditional  forms  of  subjective  evaluation  and  the  formal, 
scientific  evaluations  is  an  issue  of  major  importance  in  attempting 
to  analyze  the  role  that  evaluation  plays  in  these  organizations. 

Likewise,  the  question  of  whether  or  not  there  exists  a truly 
"scientific"  evaluation  technology  that  is  capable  of  measuring  the 
goals  of  mental  health  centers  became  problematical.  All  current 
evaluations  (both  formal  and  informal)  rely  to  a significant  extent 
on  subjective  judgments  and  are  anchored  in  particular  value  systems. 


O O 

^°For  a review  of  definitions  of  evaluation,  see  Francis  G. 
Caro,  "Approaches  to  Evaluative  Research:  A Review,  " Human 
Organization,  Vol.  28  (Summer,  1969),  pp.  87-99. 

29 

Indeed,  at  least  one  major  form  of  therapy  has  as  one  of 
its  basic  techniques  the  avoidance  of  the  universal  tendency  to  make 
evaluative  judgments.  See  Carl  R.  Rogers,  Client-Centered  Therapy 
(Boston:  Houghton  Mifflin,  1951). 
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Yet,  there  still  remains  a difference  between  what  the  author  has 
called  "formal,  scientific  evaluation"  and  purely  subjective  evaluation: 
the  former  is  public  and  amenable  to  retesting  by  people  other  than 
the  original  evaluator  while  the  latter  type  of  judgment  is  the 
"property"  of  the  evaluator.  ^ 

Thus  the  above  definition  of  evaluation  serves  to  distinguish 
between  those  centers  that  are  carrying  out  formal  scientific  evalua- 
tions and  those  which  are  not.  However,  in  order  to  more  precisely 
examine  the  role  that  evaluation  plays  in  these  organizations,  the 
following  three  dimensions  were  developed.  Among  other  things, 
they  help  to  distinguish  between  truly  critical  evaluations  and  merely 
token  evaluations: 

Token  Evaluation  Critical  Evaluation 

Scope:  Narrow  Broad 

Focus:  Technique -Oriented^  Accomplishment -Oriented 

Impact:  Symbolic^  Substantive 


30As  is  often  the  case  when  one  attempts  to  make  a distinction 
between  two  categories,  there  are  some  examples  that  do  not  fit  into 
either  category.  Such  cases  will  be  noted  in  the  discussion  concerning 
specific  evaluative  techniques  in  Chapter  II. 

31 

For  the  distinction  between  evaluations  which  focus  on  techni- 
ques and  those  which  focus  on  more  accomplishment-oriented  goals, 
see  Brian  MacMahon,  Thomas  Pugh,  and  George  Hutchison,  "Principles 
in  the  Evaluation  of  Community  Mental  Health  Programs,  " American 
Journal  of  Public  Health,  Vol.  51  (July,  1961),  pp.  964-968. 

^For  a discussion  of  the  differences  between  "symbolic"  and 
"substantive"  evaluative  research,  see  Joseph  Eaton,  "Symbolic  and 
Substantive  Evaluative  Research,  " in  Program  Evaluation  in  the  Health 
Fields,  ed.  Herbert  C.  Schulberg,  Alan  Sheldon,  and  Frank  Baker 
(New  York:  Behavioral  Publications,  1969),  pp.  506-524. 
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Community  mental  health  centers  may  undertake  evaluations 

that  conform  for  a variety  of  reasons  (to  be  explored  later)  to  the 

definition  of  formal,  scientific  evaluation  but  may  not  achieve  or 

even  attempt  to  achieve  any  real  change.  When  Wildavsky  argues 

that  an  organization  cannot  evaluate  itself,  he  is  probably  saying 

that  whatever  formal  evaluation  it  undertakes  is  likely  to  be  narrow 

in  scope,  technique-oriented,  and  unlikely  to  achieve  any  substantive 

change.  These  three  dimensions  will  be  employed  as  guidelines  in 

3 3 

our  analysis  of  the  methods  of  formal  evaluation. 

The  concept  of  organization  is  even  more  difficult  to  define 
and  use  systematically  than  evaluation.  ^ An  organization  is  viewed 
here  as  an  open  system  consisting  of  the  interaction  of  its  com- 
ponents including  the  formal  organizational  structure  and  the  informal 

q c 

relationships  developed  by  organizational  members.  ° Behavior 


33These  three  dimensions  will  be  used  as  general  guidelines 
in  our  analysis  of  each  technique.  No  attempt  was  made  to  construct 
quantitative  indices  of  any  of  these  dimensions.  For  a discussion  of 
the  utility  of  employing  such  qualitative  guidelines,  see  Anselm  Strauss, 
et  al.  , Psychiatric  Ideologies  and  Institutions  (New  York:  Free  Press, 
1964),  pp.  31-35. 

34 

A review  by  Ralph  Stodgill  found  some  eighteen  different 
major  orientations  to  the  concept  of  organization.  See  his  "Dimensions 
of  Organizational  Theory,  " in  Approaches  to  Organizational  Design,  ed. 
James  D.  Thompson  (Pittsburgh:  University  of  Pittsburgh  Press, 

1963),  pp.  1-56. 

35 

For  the  classic  analysis  of  the  distinction  between  these  two 
elements  of  an  organization,  see  Alvin  W.  Gouldner,  "Organizational 
Analysis,  " in  Sociology  Today:  Problems  and  Prospects,  ed.  Robert 
K.  Merton,  et  al  (New  York:  Basic  Books,  1959),  pp.  400-428. 
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within  organizations  is  also  shaped  by  factors  such  as  the  technology 
employed  by  organizational  members  in  the  accomplishment  of 
their  tasks,  the  personalities  of  organizational  members,  and  the 
values  which  organizational  members  hold.  ^6  Nearly  all  organi- 
zations are  strongly  influenced  by  their  environments.  Important 
elements  of  the  environments  of  these  community  mental  health 
centers  included  the  following:  (1)  the  clientele  of  the  organiza- 
tions, (2)  the  suppliers  of  the  organization,  including  the  funding 
agencies  and  private  foundations,  (3)  competitors  of  the  organiza- 
tions which  include  both  other  governmental  agencies  competing  for 
funds  and  also  private  practitioners  insofar  as  they  overlap  with  the 
centers  in  serving  the  same  clientele,  (4)  regulatory  groups  which 
include  not  only  the  funding  agencies  but  also  the  district  mental 

health  boards  which  oversee  the  operations  of  the  mental  health 
3 7 

centers. 


o o 

DFor  organizational  studies  which  tend  to  emphasize  organi- 
zational structure,  technology,  and  interpersonal  relationships 
respectively,  see  Peter  M.  Blau  and  Richard  Schoenherr,  The 
Structure  of  Organizations  (New  York:  Basic  Books,  1971);  Charles 
Perrow,  Complex  Organizations:  A Critical  Essay  (Glenview: 

Scott,  Foresman,  and  Company,  19  72);  and  Chris  Argyris, 

Integrating  the  Individual  and  the  Organization  (New  York:  John 
Wiley  and  Sons,  1964). 

37 

This  discussion  of  the  elements  of  the  organizational  environ- 
ments is  based  on  the  analysis  of  James  D.  Thompson,  Organizations 
in  Action  (New  York:  McGraw-Hill,  1967),  p.  28. 
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Organizational  Theory 

The  purpose  of  this  dissertation  is  to  analyze  the  role  of 
evaluation  in  this  group  of  organizations  and  examine  the  issue  of 
whether  or  not  organizations  can  critically  evaluate  themselves. 
However,  research  into  these  problem  areas  revealed  a closer  inter- 
relationship between  them  and  some  important  areas  of  research 
in  the  bodies  of  the  literature  of  organizational  theory  and  public 
administration  including  the  following:  (1)  the  issue  of  survival  versus 
goal-oriented  models  of  organizations,  (2)  the  relationship  between 
the  "natural"  and  "artificial"  components  of  organizations,  (3)  the 
technology  employed  by  organizations,  (4)  the  issues  of  accountability 
and  responsibility  of  public  organizations. 

The  issue  of  evaluation  in  organizations  is  related  to  the 
problem  of  whether  or  not  organizations  should  be  viewed  as  goal- 

O O 

oriented  or  survival-oriented  systems.  J Important  questions  are 
rarely  stated  as  simply  as  this,  because  the  accomplishment  of 
goals  may  contribute  to  the  survival  of  the  organization,  or  the 
organization  may  be  goal-oriented  but  to  goals  other  than  those  that 
were  the  basis  for  which  the  organization  was  established.  Also, 
one  of  the  major  obstacles  to  evaluation  may  be  the  lack  of  clarity 

q p 

°°See  Amitai  Etzioni,  "Two  Approaches  to  Organizational 
Research,  " Administrative  Science  Quarterly,  Vol.  5 (June,  1960), 
pp.  257-278. 
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of  the  goals  of  the  organizations. 


Following  the  argumenFof  Mohr,  however,  we  seek  to  clarify 
the  degree  to  which  evaluation  is  undertaken  to  satisfy  externally 
oriented  goals  or  internally-oriented  goals.  Distinguishing 
between  the  internally  and  externally  oriented  goals  (or  survival 
andgoal-oriented  systems)  thus  reduces  itself  to  the  difficult  problem 
of  assigning  motives.  In  order  to  aid  in  this  task,  the  following 
guidelines  were  devised  based  on  "predictions"  from  the  goal  and 
survival  models:  . rrgar.izs-.  r:  - 

- - (1)  Impetus:  A basic  desire  to  measure  the  degree  of 

their  success  in  achieving  their  goals  would  be  an  important 
and  natural  component  of  an  organization  which  corresponds 
to  the  goal-oriented  model  of  an  organization.  The  only 
1:  impetus  which  a survival-oriented  organization  would  find 

compelling  to  undertake  evaluation  would  be  that  it  would 
lead  to  the  acquisition  of  funds,  whether  because  evaluation 
is  required  by  law  or  because  it  can  be  used  as  a positive 
rrr.:  argument  in  obtaining  them. 

c:  (2)  Type  of  Evaluation:  A truly  critical  self-evaluation 

carried  out  by  an  organization  would  be  likely  to  be  wide  in 
r:  scope,  focus  on  the  end  goals  of  the  organization,  and  result 

in  extensive  substantive  change.  A survival -oriented  organiza- 
- : tion  would  be  likely  to  measure  only  those  goals  which  it  is 

required  to,  or  measure  those  which  make  it  look  good. 


*^See,  e.  g. , O.  Ruth  McQuown,  From  National  Agency  to 
Regional  Institution:  A Study  of  TVA  in  the  Political  Process 
(Ph.  D.  dissertation.  University  of  Florida,  1961). 

— ^Lawrence  B.  Mohr,  "The  Concept  of  Organizational  Goals,  " 
American  Political  Science  Review,  Vol.  67  (June,  1973),  pp.  470- 

481..  . ...  _ 
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A second  problem  area  to  which  the  issue  of  the  role  of 
evaluation  in  organizations  is  related  concerns  the  relationship 
between  the  natural  and  artificial  components  of  an  organization. 

In  a classic  article,  Alvin  Gouldner  notes  that  organizational 
theorists  tended  to  focus  either  on  the  planned,  "rational"  components 
of  an  organization  (e.  g.  , its  formal  authority  structure)  or  the 
spontaneous,  unplanned  components  such  as  sentiments,  beliefs, 
unprescribed  social  structures,  personal  enmities  and  friendships.  41 
Both  aspects  of  organization  are  important  and  particular 
emphasis  is  given  here  to  studying  the  interaction  between  the  two. 

It  would  seem  likely  that  formal,  scientific  evaluation  is  likely  to 
encroach  on  areas  which  were  previously  part  of  the  unplanned, 
natural  system  aspects  of  an  organization.  Indeed,  this  probability 
is  the  basis  of  much  of  Wildavsky's  argument  as  to  why  evaluation 
cannot  succeed;  the  evaluators  will  run  into  opposition  from  these 
informal  yet  crucial  aspects  of  organizations.  All  organizations  have 
both  components  or  "systems.  " In  attempting  to  successfully 
introduce  formal  evaluation  into  an  organization,  an  evaluator  may 


41  Gouldner  has  pointed  out  that  one  problem  with  using  the 
term  "natural  system"  is  that  it  is  not  clearly  definable  but  is  used 
as  a residual  category  for  all  of  the  unprogrammed  elements  of 
an  organization.  However,  until  someone  develops  a more  empirically 
based  typology  of  these  "natural  system  elements,  " we  are  forced 
to  use  the  term  anyway.  See  Gouldner,  "Organizational  Analysis,  " p.  410 


% 
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emphasize  either  "artificial"  components  of  the  organization,  such 

as  authority  and  technology,  or  informal,  natural  system  elements 

such  as  personal  friendships  with  the  other  staff  of  the  organization. 

Many  students  of  organizations  believe  that  the  technology 

employed  by  an  organization  is  the  single  most  important  factor 

42 

affecting  behavior  within  organizations.  Technology  has  been 

defined  in  many  ways  with  some  writers  using  it  to  refer  to  specific 

modes  of  production.  However,  here  the  author  is  using  it  to  mean: 

(1)  the  degree  of  completeness  of  knowledge  concerning  cause  and 

effect,  (2)  the  degree  to  which  there  are  crystallized  standards  of 
43 

desirability.  Taken  together,  these  two  factors  would  seem 

likely  to  be  important  in  determining  the  reception  of  the  evaluation 

by  the  personnel  of  the  organization  in  which  it  is  to  be  carried  out. 

If  there  is  a great  deal  of  consensus  concerning  desirable  standards 

and  relationships  between  cause  and  effect,  an  evaluation  should 

have  a great  deal  of  legitimacy  and  encounter  less  resistance  and 

different  forms  of  resistance,  if  any  (e.g.  , cheating  rather  than  open 
, 44 

challenge). 


42See,  e.  g.  , Charles  Perrow,  "Some  Reflections  on  Technology 
and  Organizational  Analysis,  " in  Modern  Organizational  Theory,  ed. 

Anant  R.  Negandhi  (Kent:  Kent  State  University  Press,  1973),  pp.  48-54. 

43 

This  is  based  on  the  analysis  of  James  D.  Thompson, 
Organizations  in  Action,  pp.  86-92. 

^Thus,  this  work  examines  the  perceptions  held  by  members 
of  the  organizations  concerning  the  validity  of  the  technology  of 
evaluation.  The  legitimacy  of  an  evaluative  technique  refers  to  the 

extent  to  which  it  is  perceived  as  being  both  valid  and  reliable. 
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However,  the  technology  of  the  mental  health  field  would 
appear  to  fall  in  the  category  of  technologies  where  there  is  relatively 
little  consensus  concerning  cause  and  effect  or  desirable  standards. 

In  such  a situation,  there  would  appear  to  be  a great  potential  for  skepti- 
cism and  resistance  to  a formal,  scientific  evaluation.  The  technolog- 
ically oriented  organizational  theorist  would  predict  that  a truly 
developed  evaluation  technology  would  be  able  to  overcome  such  resist- 
ance. Others  would  see  technology  as  being  a less  important  obstacle 
to  formal  evaluation  than  the  complex  set  of  norms,  beliefs,  friend- 
ships, and  cliques  that  we  call  the  natural  system  of  the  organization. 
This  dissertation  explores  the  extent  to  which  resistance  to  formal 

evaluation  is  based  on  skepticism  concerning  the  technology  of  evalua- 

46 

tion,  natural  systems  factors,  or  other  reasons. 


4^Thus,  there  are  not  only  a large  number  of  different  types  of 
therapy  but  also  a disagreement  concerning  underlying  models  of 
treatment  such  as  the  "medical  model"  and  the  "public  health  model.  " 

For  a discussion  of  these  issues,  see  Jerome  D.  Frank,  "The  Be- 
wildering World  of  Psychotherapy,  " Journal  of  Social  Issues,  Vol.  28, 

No.  4 (1972),  pp.  27-44,  and  Louis  D.  Cohen,  "Health  and  Disease: 
Observations  on  Strategies  for  Community  Psychology,  " in  Issues  in 
Community  Psychology  and  Preventive  Mental  Health,  American  Psycholo- 
gical Association  (New  York:  Behavioral  Publications,  1971),  pp.  55-74. 

^®A  distinction  is  made  between  the  technology  of  the  therapies 
employed  by  the  personnel  of  these  organizations  and  the  technology 
of  the  evaluations  used  to  evaluate  the  effectiveness  of  these  therapies. 

For  a discussion  of  the  problems  of  current  evaluative  techniques,  see 
Gerald  Caplan,  "The  Nature  and  Problems  of  Evaluation  in  Community 
Mental  Health,  " in  Comprehensive  Mental  Health:  The  Challenge 
of  Evaluation,  ed.  Leigh  M.  Roberts  et  al.  (Madison:  University  of 
Wisconsin  Press,  1968),  pp.  3-14. 
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While  the  major  focus  of  this  dissertation  is  on  the  concept 
of  internal  organizational  evaluation,  the  answer  to  the  problem  of 
how  critically  organizations  evaluate  their  own  activities  has 
important  implications  for  the  issue  of  accountability  in  government. 

A major  text  in  public  administration  distinguishes  between  responsi- 
bility (responsiveness  to  other  people's  values)  and  accountability 
(the  methods,  procedures  and  forces  that  determine  what  values 
will  be  reflected  in  administrative  decisions).  Thus  accountability 
is  the  method  (or  methods)  used  to  enforce  the  responsiveness  of 
public  organizations  to  the  values  of  the  public  and  its  representatives. 
The  implicit  assumption  behind  most  administration,  and  particularly 
in  the  crucial  area  of  budgeting,  is  that  agencies  will  not  critically 
evaluate  themselves  and  that  responsiveness  to  the  values  of  the 
public  is  maintained  only  by  the  review  of  performance  of  the  organi- 
zation by  someone  outside  of  the  organization. 

The  budgeting  system  is  now  basically  incremental  with  little 
pretense  on  the  part  of  the  legislators  of  being  able  to  judge  the 
comparative  worth  of  any  agency's  operation's.  48  If  organizations 

“^Herbert  A.  Simon  et  al.  , Public  Administration  (New 
York:  Alfred  A.  Knopf,  1950),  p.  513.  This  distinction  is  an 
analytical  one  because  most  of  the  persons  interviewed  to  use  term 
"accountability"  to  cover  both  the  methods  of  evaluation  and  the 
responsiveness  obtained. 

48 

See  Aaron  Wildavsky,  The  Politics  of  the  Budgetary  Process 
(Boston:  Little,  Brown,  and  Company,  1964). 
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were  capable  of  and  willing  to  provide  to  legislators,  funding  agencies, 

and  their  local  communities  information  which  gives  a true  and 

objective  assessment  of  how  well  they  are  achieving  their  goals 

(as  defined  by  the  public  and  their  clientele),  the  budgetary  and 

political  process  would  be  radically  altered. 

Indeed,  the  development  of  scientific  evaluation  is,  in  part, 

the  attempt  to  provide  an  alternative  to  "political  rationality.  " 

Thus,  it  may  be  seen  as  an  attempt  to  provide  the  equivalent  of  the 

market  mechanisms  of  competition  and  profit  which  supposedly 

49 

provide  objective  criteria  of  worth  to  private  organizations. 

Indeed,  many  writers  feel  that  most  of  the  differences  between  public 
and  private  organizations  relate  directly  or  indirectly  to  this  problem 
of  the  lack  of  objective  criteria  to  judge  the  efficiency  and  effective- 
ness of  public  organizations  as  the  following  list  would  suggest: 

(1)  Public  organizations  are  controlled  more  by  superiors 
than  by  the  price  system. 

(2)  Public  organizations  are  based  on  a type  of  funding 
that  is  highly  contingent  upon  the  previous  experience  and 
perceptions  of  superiors. 

(3)  Public  organizations  tend  to  have  more  vaguely  defined 
or  multiple  goals  among  which  cost  reduction  does  not  receive 
very  high  priority. 


4^0f  course,  how  closely  even  private  organizations  correspond 
to  the  "ideal"  economic  model  is  open  to  debate  with  some  studies 
demonstrating  the  equivalent  of  a "political  mechanism"  in  private 
organizations.  See,  e.  g.  , Richard  Cyert  and  James  G.  March,  A 
Behavioral  Theory  of  the  Firm  (Englewood  Cliffs:  Prentice  Hall, 
1963). 
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(4)  Public  organizations  are  relatively  insulated  from 
automatic  penalties  and  rewards  of  the  price  system. 

(5)  Public  organizations  lack  objective  tests  of  efficiency 
of  product  or  service. 

(6)  Public  organizations  are  able  at  times  to  shift  their 
costs  to  other  agencies  rather  than  face  them  or  go  under. 

(7)  Public  organizations  are  not  generally  provided  by 
cost  reduction  with  opportunity  for  growth.  50 

Most  of  the  attempts  or  movements  to  improve  the  quality 

of  the  output  of  government  revolve  around  either  a return  to  the 

"economic  model"  or  a turn  to  the  increased  use  of  scientific 

Cl 

measures  of  the  effectiveness  of  public  organizations.  A few 
writers  have  defended  the  current  system  of  allocation  of  resources 

CO 

which  is  dominated  by  political  judgments  and  incrementalism. 

This  study  explores  the  factors  which  help  internal  evaluators  achieve 

responsiveness  to  the  goals  of  their  evaluations  and  the  conditions 

under  which  internal  organizational  evaluations  are  likely  to  provide 

the  "functional  equivalent"  of  external  forms  of  accounting  for  the 

53 

purposes  of  achieving  responsibility  of  public  organizations. 


C(] 

Gary  L.  Wamsley  and  Mayer  N.  Zald,  The  Political  Economy 
of  Public  Organizations  (Lexington:  D.  C.  Heath  and  Company,  1973), 
pp.  5-6.  Of  course,  these  are  tendencies  rather  than  absolute  differences. 

51 

Milton  Friedman  has  been  one  of  the  major  proponents  of 
returning  governmental  functions  to  the  private  market.  See  his  book. 
Capitalism  and  Freedom  (Chicago:  University  of  Chicago  Press,  1962). 

52 

See,  e.  g.  , Charles  E.  Lindblom,  The  Intelligence  of 
Democracy:  Decision-Making  Through  Mutual  Adjustment  (New 
York:  The  Free  Press  of  Glencoe,  1965). 

53 

The  use  of  the  term  function  and  other  terms  related  to  it 
follows  the  usage  of  Robert  Merton.  See  Robert  Merton,  Social  Theory  ’ 
and  Social  Structure  (2nd  ed.  ; Glencoe:  Free  Press,  1957),  pp.  19-84. 
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The  concept  of  professionalism  is  also  related  to  the  issues 

of  evaluation  and  accountability.  The  mental  health  organizations 

visited  in  this  study  are,  to  a large  extent,  composed  of  persons  who 

are  classified  as  professionals,  including  clinical  psychologists, 

54 

psychiatrists,  and  psychiatric  social  workers.  One  of  the 
characteristics  that  distinguishes  professional  occupations  is  their 
claim  to  be  self-policing  concerning  the  quality  of  the  services 
offered  by  members  of  the  profession:  "The  profession  should  be 
accepted  as  the  final  arbiter  in  any  disputes  over  the  validity  of  any 
technical  solution  lying  within  its  area  of  supposed  competence. 

This  statement  would  suggest  that  the  newer  forms  of  formal, 
scientific  evaluation  would  be  regarded  as  an  impingement  upon  the 
more  traditional  form  of  professional  evaluation- -an  issue  that  is 
explored  here.  ^ 

Hypotheses  and  Theoretical  Frameworks 
Howard  Becker  has  argued  that  it  is  useful  for  individuals 
carrying  out  research,  particularly  research  with  a great  deal  of 


54jr0r  a discussion  of  the  criteria  by  which  occupations  are 
classified  as  professions,  see  William  J.  Goode,  "The  Theoretical 
Limits  of  Professionalization,  " in  The  Semi-Professions  and  Their 
Organization,  ed.  Amitai  Etzioni  (New  York:  Free  Press,  1969), 
pp.  266-314. 

55 

Ibid.  , p.  278. 

r g 

°°For  a discussion  of  the  alleged  lack  of  accountability  and 
responsibility  of  professionals,  see  Elliot  Freidson,  Professional 
Dominance  (New  York:  Atherton,  1970). 
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qualitative  material,  to  give  a "natural  history"  of  the  course  of 
their  work  including  the  selection  of  their  approaches  and  hypotheses.  ^ 
The  major  focus  of  this  research  has  remained  constant  and  is  to 
explore  the  role  that  evaluation  has  played  in  these  organizations 
with  a particular  view  to  examining  the  argument  that  organizations 
cannot  evaluate  themselves  due  to  their  strong  attachments  to 
survival  needs.  Stated  in  hypothetical  form,  this  thesis  can  be 
summarized  as  follows: 

Due  to  survival  needs,  organizations  cannot  critically 
evaluate  themselves,  and  any  evaluations  which  they  do 
implement  will  tend  to  be  narrow,  technique -oriented, 
and  symbolic. 

Although  the  above  hypothesis  is  used  to  focus  the  analysis 
of  the  role  of  evaluation,  this  research  is  exploratory  and  not  directed 
at  the  rigorous  testing  of  hypotheses.  Rather,  this  study  is  more 
inductive  than  deductive.  58  The  research  revealed  several  relation- 
ships among  variables  related  to  evaluation  and  organizations. 

These  relationships  are  formally  stated  as  hypotheses  at  the  conclusion 
of  each  chapter.  However,  since  these  propositions  were  developed 
only  after  the  beginning  of  the  research,  there  is  no  pretense  that 

V7 

'Howard  S.  Becker,  Sociological  Work:  Method  and  Sub- 
stance (Chicago:  Aldine  Publishing  Company,  1970),  p.  69. 

5 8 

For  an  argument  that  the  social  sciences  should  constantly 
"mix"  induction  and  deduction  in  research,  see  Heinz  Eulau,  Micro- 
Macro  Political  Analysis  (Chicago:  Aldine  Publishing  Company,  1969), 

p.  352. 


24 


they  have  been  tested  here. 

As  far  as  theoretical  frameworks  are  concerned,  this  research 
employs  insights  derived  from  bodies  of  literature  of  organizational 
theory  and  public  administration  described  above.  For  the  purposes 
of  elegance  and  simplicity,  it  is  often  viewed  as  desirable  to  fit 
research  into  a single  theoretical  framework.  While  such  an  "integrating 
theory"  is  lacking  here,  all  four  problem  areas  of  organizational 
theory  are  closely  interrelated  as  employed  here.  Thus,  a critically 
self-evaluating  organization  would  be  able  to  provide  accountability 
to  the  public  without  outside  intervention  and  have  overcome  the 
problem  of  the  technical  adequacy  of  the  technology  of  evaluation. 

Such  an  organization  would  also  have  conquered  resistance  from  the 
natural  system  and  resisted  the  tendency  of  organizations  to  be 
dominated  purely  by  survival  needs. 

Methodology 

<r 

Four  kinds  of  primary  data  were  obtained  in  this  study 
of  these  fourteen  organizations:  (1)  participant  observation  data, 

(2)  semi-structured  interviews  with  directors,  clinicians,  and 
evaluators  of  these  organizations,  (3)  questionnaires  from  these 

5®Robert  Merton  has  noted  the  curiosity  of  "post  factum" 
research  in  which  the  evidence  always  supports  the  propositions 
since  the  latter  were  formulated  after  the  research  was  concluded. 

See  Robert  Merton,  Social  Theory  and  Social  Structure,  p.  93. 
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groups  of  individuals,  (4)  documents  from  these  and  other  mental 

health  organizations  related  to  evaluation.  The  advantage  of  using 

more  than  one  technique  of  data  collection  is  becoming  increasingly 

60 

recognized  in  research.  Propositions  that  are  supported  by  more 

than  one  method  of  observation  are  more  likely  to  be  valid  than  those 

61 

supported  by  only  one  method. 

The.  major  portion  of  the  participant  observation  consisted 
of  five  months  of  observation  of  a community  mental  health  center 
in  which  meetings  were  attended.  Evaluation,  budgets,  organizational 
goals,  and  philosophy --all  were  considered  and  raised  sometimes 
heated  discussion,  perhaps  because  of  the  youth  and  flux  of  this 
organization  which  was  less  than  a year  old  when  observation  first 
began.  One  expert  has  noted  that  organizations  which  are  new  or 
are  undergoing  stressful  situations  (e.  g.  , budget  cuts)  provide 

especially  fruitful  material  in  which  to  view  the  dynamics  of 

• 62 
organizations. 

The  author  was  also  a participant  observer  at  meetings  of 
a consortium  of  mental  health  organizations  which  had  been  formed 

®^See,  e.  g.  , Eugene  J.  Webb  et  al.  , Unobtrusive  Measures: 
Nonreactive  Research  in  the  Social  Sciences  (Chicago:  Rand  McNally, 
1966),  pp.  1-28. 


62 

Robert  Bogdan,  Participant  Observation  in  Organizational 
Settings  (Syracuse:  Syracuse  University  Press,  1972),  p.  32. 
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to  study  and  foster  evaluation.  These  meetings  were  particularly 
useful  because  they  were  attended  by  representatives  of  NIMH  and 
the  State  Division  of  Mental  Health  of  the  State  of  Florida.  Meetings 
held  by  a mental  health  district  board  were  also  observed  by  this 
author.  Finally,  a staff  meeting  was  attended  at  a second  center 
in  which  issues,  complaints,  and  general  perspective  were  strikingly 
similar  to  those  at  the  original  center. 

Participant  observation  is  particularly  useful  in  studying 

concepts  which  tend  to  be  viewed  as  socially  desirable.  Few  persons 

are  likely  to  admit  to  being  "against  evaluation.  " Extended  observation 

is  almost  certainly  less  likely  to  suffer  from  such  a bias  because  it 

is  difficult  for  an  individual  to  hide  his  true  feelings  in  his  actual 

63 

place  of  work  over  an  extended  period  of  time. 

The  interviews  followed  a form  outlined  by  Merton  and  others 
in  which  certain  basic  problem  areas  are  explored  in  depth  but  with 
the  flexibility  to  follow-up  on  useful  leads  or  to  abandon  obviously 

n i 

"dead"  lines  of  questioning.  Major  issues  explored  in  the  interviews 
included  the  following  topics:  (1)  the  role  of  the  evaluator,  (2)  the 
methodology  of  evaluation,  (3)  the  function  of  evaluation,  (4)  the 
problems  of  implementing  evaluation,  (5)  the  impact  of  evaluation 
on  the  organization. 

®^See  Becker,  Sociological  Work,  p.  69. 

64 

See  Robert  Merton  et  al.  , The  Focused  Interview  (Glencoe: 
Free  Press,  1956). 
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The  total  number  of  interviews  conducted  was  thirty-five  with 
forty-seven  different  individuals  in  these  thirteen  centers  and  one 
clinic.  Interviews  were  carried  out  with  three  major  groups  of 
individuals:  directors,  clinicians,  and  evaluators.  Not  all  of  the 
organizations  had  an  "evaluator.  Comparisons  are  made  between 
those  organizations  which  were  relatively  interested  in  evaluation 
so  as  to  invest  in  an  evaluator  (or  evaluators)  and  those  which  did 
not  make  this  investment. 

The  interviews,  with  two  exceptions,  were  taped,  which  makes 

it  possible  to  present  the  comments  of  these  clinicians,  directors, 

fi  fi 

and  evaluators  in  their  own  words.  It  was  found  that  the  months 
of  participant  observation  enabled  the  author  to  ask  far  sharper 
questions  than  would  have  been  otherwise  possible,  as  well  as  making 
it  possible  to  supplement  abstract  questions  with  specific  examples. 
This  tactic  was  particularly  useful  when  interviewees  hesitated  in 
answering  delicate  questions  concerning  their  own  organizations 
but  would  not  hesitate  to  answer  frankly  the  same  kind  of  question 
when  it  concerned  a situation  that  had  been  put  in  hypothetical  form 

6 fi 

During  the  study,  five  of  the  organizations  had  at  least 
one  person  devoted  specifically  to  evaluation.  Some  of  the  others 
were  considering  the  hiring  of  an  evaluator. 

fi  fi 

DFor  a discussion  of  the  effect  taping  interviews  has  on 
responses,  see  Raymond  L.  Gorden,  Interviewing:  Strategy, 
Techniques,  and  Tactics  (Homewood:  Dorsey  Press,  1969), 
pp.  175-178. 
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fi  7 

or  illustrated  from  another  center. 

The  questionnaire  was  constructed  in  order  to  provide  an 
indicator  of  how  favorable  the  different  groups  (i.  e.  , clinicians, 
directors,  and  evaluators)  were  towards  the  idea  of  formal  evaluation. 
It  was  purposefully  kept  much  shorter  than  it  would  have  been  if  it 
had  sought  to  obtain  all  the  information  that  was  potentially  relevant. 

This  course  was  suggested  by  the  open  hostility  of  members 
of  these  mental  health  organizations  towards  the  rapidly  increasing 
amount  of  paperwork  to  which  they  had  been  subjected,  including 
numerous  questionnaires  from  various  sources.  The  implication 
was  that  a lower  rate  of  return  could  be  expected  if  the  questionnaire 
was  long  and  involved.  One  expert  has  argued  that  the  dangers  of 
bias  from  a low  rate  of  return  are  more  serious  than  the  sacrifice 
of  information  that  results  from  a shortened  questionnaire.  68 


fi  7 

Sam  Sieber  describes  similar  advantages  resulting  from 
the  combination  of  participant  observation  and  other  forms  of  data 
such  as  surveys  and  interviewing.  See  Sam  Sieber,  "The  Integration 
of  Fieldwork  and  Survey  Methods,  " American  Journal  of  Sociology, 

Vol.  78  (May,  1973),  pp.  1335-1359. 

6 8 

See  A.  Bradford  Hill,  "Scientific  Method  in  Field  Surveys: 
General  Principles  of  Field  Surveys"  in  Proceedings  of  a Conference 
held  from  March  29  to  March  31,  1950,  Great  Britain  Medical  Research 
Council  (London:  His  Majesty's  Stationery  Office,  1951),  pp.  7-13. 
Sixty-nine  of  eighty-four  questionnaires  were  returned,  although 
four  had  to  be  discarded  because  they  had  not  been  completed. 
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Plan  of  the  Dissertation 

In  the  following  chapter,  we  shall  examine  the  formal  evaluations 
currently  being  carried  out  in  the  fourteen  organizations  under  study. 
Each  major  evaluative  technique  will  be  systematically  examined 
through  the  employment  of  the  dimensions  of  scope,  focus,  and  impact 
previously  developed  as  well  as  other  considerations.  In  Chapter  III 
attitudes  towards  formalized  evaluations  will  be  studied  and  compared 
with  attitudes  towards  more  subjective  forms  of  evaluation.  Particular 
attention  will  be  given  to  the  problems  of  resistance  to  formal 
evaluation  from  the  "natural  system"  of  the  organization  and  skepticism 
concerning  the  technology  of  formal  evaluation.  In  Chapter  IV,  we 
will  focus  upon  the  relationship  between  the  survival  needs  of  the 
organizations  and  the  decision  to  invest  in  evaluation.  Differences 
among  the  organizations  with  respect  to  their  interest  in  formal 
evaluation  will  be  analyzed.  In  Chapter  V,  we  will  examine  the  relation 
ship  between  evaluation  and  accountability  and  responsibility.  Special 
attention  will  be  given  to  the  factors  which  increase  the  likelihood 
that  evaluators  will  achieve  the  goals  of  their  evaluation  by  employing 
comparisons  between  internal  and  external  evaluations.  In  Chapter  VI, 
the  major  conclusions  will  be  reviewed  including  the  inductive 
propositions  which  emerge  from  the  study. 


CHAPTER  II 


THE  METHODOLOGY  AND  IMPACT  OF  EVALUATION 

The  purpose  of  this  chapter  is  to  analyze  the  methodology 
and  impact  of  each  formal  evaluative  technique  that  has  been 
introduced  in  these  fourteen  organizations.  To  aid  in  the  analysis, 
the  three  dimensions  outlined  in  the  last  chapter  are  employed: 

Token  Evaluation  Critical  Evaluation 

Scope:  Narrow  Broad 

Focus:  Technique -Oriented  Accomplishment -Oriented 

Impact:  Symbolic  Substantive 

"Scope"  refers  to  the  extent  to  which  the  evaluation  covers 
the  fulLrange  of  services  offered  by  the  organizations.  For  example, 
the  evaluation  may  cover  the  activities  of  only  one  particular  service 
such  as  the  outpatient  clinic,  or  it  may  include  the  full  range  of 
services  offered  including  the  inpatient,  outpatient,  emergency, 
partial,  and  educational  programs.  However,  evaluations  can  be 
too  broad  as  well  as  too  narrow.  In  such  cases,  they  are  too  vague 
to  be  able  to  yield  useful  information  about  the  quality  of  specific 
programs  and  individual  therapists. 

The  second  dimension,  "focus,  " concerns  the  nature  of  the 
goals  which  the  evaluation  measures.  For  example,  an  evaluation 
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may  focus  upon  the  extent  to  which  service  in  a particular  program 
corresponds  to  accepted  practice  as  defined  by  the  supervisory 
staff  of  the  organization.  Yet,  whether  therapy  which  is  regarded 
as  practicing  good  "techniques"  actually  produces  desirable  "end 
goals"  such  as  client  satisfaction  and  behavioral  changes  is 
problematical.  * Thus,  the  focus  of  an  evaluation  reveals  the  extent 
to  which  the  evaluation  is  based  upon  assumptions  concerning  the 
relationship  between  "intermediate"  goals  and  the  ultimate  goals 
of  the  service. 

However,  there  are  also  important  questions  concerning  what 

the  end  goals  of  these  mental  health  organizations  actually  are. 

In  a free  market  model,  the  satisfaction  of  the  consumer  is  the  major 

criterion  of  all  exchanges  of  goods  and  services  because  goods  and 

services  will  be  consumed  only  to  the  point  where  the  marginal 

utility  derived  from  them  is  at  least  equal  to  that  which  could  be 

2 

derived  from  other  goods  and  services.  Thus,  the  consumer's 
willingness  to  purchase  the  product  or  service  is  proof  by  itself  that 
the  product  is  worthwhile. 


^Thus,  when  techniques  or  "intermediate  goals"  are  focused 
upon,  a whole  series  of  assumptions  and  linkages  may  have  to  be 
fulfilled  if  the  evaluation  is  to  have  its  intended  effect.  See  O.  L. 
Deniston,  I.  M.  Rosenstock,  and  V.  A.  Getting,  "Evaluation  of 
Program  Effectiveness,  " Public  Health  Reports,  Vol.  83  (April, 
1968),  pp.  323-335. 

2 

See  Paul  A.  Samuelson,  Economics  (6th  ed.  ; New  York: 
McGraw-Hill  Book  Company,  1964),  pp.  432-433. 
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With  respect  to  public  goods,  however,  the  consumer  is  not 
paying  the  entire  cost  of  the  services  he  receives.  Likewise,  the 
community  mental  health  movement  began  with  goals  such  as  reducing 
state  mental  hospital  loads  and  the  prevention  of  such  behaviors 
as  alcoholism  and  drug  addiction.  In  fact,  the  goals  of  the  client 
of  a mental  health  center  may  be  directly  opposed  to  the  goals  of 
significant  members  of  local  communities.  Such  conflict  occurs 
frequently  when  mental  health  organizations  become  involved  in 
activities  related  to  law  enforcement  and  social  control.  ^ 

The  final  dimension  of  "impact"  is  the  most  critical  single 
dimension.  It  indicates  whether  these  organizations  take  evaluation 
seriously  enough  to  implement  changes  in  their  operations  based 
on  the  recommendations  resulting  from  the  evaluation.  However, 
many  of  the  evaluations  have  only  been  recently  introduced. 3  4 5 Also, 
no  systematic  accounting  of  the  impact  of  evaluation  was  carried  out 
within  the  organizations  in  which  it  had  been  introduced.  ^ As  a 


3 ? 

See  Allan  Beigel,  "Law  Enforcement,  the  Judiciary,  and  Mental 
Health:  A Growing  Partnership,  " Hospital  and  Community  Psychiatry, 
Vol.  24  (September,  1973),  pp.  605-609. 

4An  important  qualification  of  the  conclusions  reached  in 
this  chapter,  then,  concerns  the  time  period  in  which  this  study  was 
carried  out.  In  particular,  the  impact  of  the  evaluation  might  be 
found  to  be  different  if  undertaken  at  a later  point  in  time.  This 
point  will  be  discussed  further  in  the  next  chapter. 

5 

The  fact  that  these  organizations  had  not  planned  to  assess  the 
impact  of  the  evaluation  in  a systematic  way  is  interesting  in  itself 
because  of  the  costs  of  the  evaluations.  It  is  also  to  be  discussed  later. 
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consequence  of  the  above  reasons,  interviews  are  used  here  as  the 
basic  source  of  information  concerning  the  impact  that  each 
evaluation  has  had. 

Table  2.1  summarizes  the  major  categories  of  evaluative 
techniques  which  had  already  been  introduced  or  were  in  the  planning 
stage  at  the  time  of  this  study.  All  meet  the  criteria  of  formal, 
scientific  evaluation  introduced  in  the  first  chapter. 


TABLE  2.  1 

SUMMARY  OF  EVALUATIONS 


Consumer 

Satisfaction 

Surveys 

Goal 

Attainment 

Models 

Needs 

Assessment 

Surveys 

Problem - 

Oriented 

Records 

Introduced 

4 

3 

3 

1 

Planned 

5 

4 

1 

1 

Total 

9 

7 

4 

2 

In  addition  to  the  above  categories  of  evaluation,  there  were 
several  other  techniques  which  either  were  confined  to  one  organiza- 
tion or  were  on  the  borderline  as  to  whether  they  met  the  criteria 
of  formal,  scientific  evaluation.  Each  of  these  methods  is  examined 
with  respect  to  the  three  dimensions  of  scope,  focus,  and  impact. 


As  noted  previously,  our  most  basic  distinction  between 
scientific  and  nonscientific  evaluations  is  whether  they  are  open  to 
retesting  by  someone  other  than  the  original  evaluator  (or  evaluators). 
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Additional  considerations  which  are  relevant  to  the  problem  of  the 
role  of  evaluation  are  also  noted. 

Consumer  Satisfaction  Surveys 

Table  2. 1 reveals  that  the  single  most  popular  evaluative 
technique  employed  in  these  organizations  is  the  consumer  satisfaction 
survey.  These  surveys  generally  measure  an  individual's  overall 
reaction  to  the  service  he  receives.  An  example  of  one  such  survey 
in  Table  2.2  shows  that  extremely  broad  questions  are  asked.  These 
questions  do  not  discriminate  between  satisfaction  with  different 
programs  much  less  individual  therapists.  This  lack  of  discrimination 
makes  it  difficult  to  make  specific  recommendations  from  this  kind 
of  survey.  The  dissatisfaction  of  an  individual  client  may  be  a 
generalized  displeasure  towards  the  entire  center  but  it  may  also 
be  resentment  at  a particular  program,  individual  therapist,  or  the 
attitudes  of  the  receptionists. 

Thus,  consumer  satisfaction  surveys  usually  provide  information 
that  is  too  broad  in  scope  to  be  useful  in  making  decisions.  These 
surveys  can  be  made  more  specific  in  order  to  acquire  more  useful 
information.  However,  even  these  very  short  and  simple  surveys 
get  generally  low  rates  of  return  from  the  clients  to  whom  they  are 
sent.  Centers  expect  to  receive  only  about  15  to  2 5 percent  of  the 
questionnaires  mailed  out.  Furthermore,  a large  percentage  of  the 
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TABLE  2.  2 

EXAMPLE  OF  A CONSUMER  SATISFACTION  SURVEY 


(1)  Did  you  have  any  problems  getting  service  at 

Mental  Health  Center?  Yes No 

(2)  Were  there  any  services  you  felt  you  should  have  received  and 

didn't?  Yes No 

(3)  Were  you  satisfied  with  the  services  you  received? 

Very  Satisfied  

Satisfied  

Indifferent  

Dissatisfied  

Very  Dissatisfied  

(4)  Would  you  return  to  the  Mental  Health  Center  if  you  felt  a 

need  for  further  services?  Yes No 

(5)  Do  you  feel  different  about  your  problems  now? 

Much  Better  

Better  

Same  

Worse  

Much  Worse  

(6)  Do  you  attribute  this  to  the  treatment  you  received  at  the  Center? 

Yes  Partly No 

Source:  Adapted  from  "Cumulative  Report  on  Patient  Satisfaction  with 
the  Mental  Health  Center,  " P.  E.  P.  Newsletter  Compendium 
(Minneapolis:  Program  Evaluation  Project,  n.  d.  ),  p.  13. 
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clientele  of  most  centers  are  relatively  uneducated  and  do  not 
understand  complex  questions.  In  short,  with  a more  detailed  survey 
the  evaluators  would  risk  an  even  smaller  and  more  biased  sample 
of  responses  than  they  presently  get. 

Certain  organizations  used  interviewers  to  gather  part  or  all 

data  for  the  consumer  satisfaction  surveys.  Using  interviewers 

tends  to  sharply  increase  the  percentage  of  clients  responding  to 

the  survey  but  it  also  greatly  increases  the  cost.  The  problem  of 

securing  useful  information  from  consumer  satisfaction  surveys 

involves  finding  the  proper  scope  and  a representative  sample.  Both 

of  these  objectives  must  be  compromised  because  of  the  urgent  problem 

of  scarcity  of  resources.  Therefore,  most  of  the  organizations  took 

the  option  of  having  a very  general  set  of  questions  administered 

through  the  mail  despite  the  problems  of  interpretation  and  bias 

7 

that  result  from  this  type  of  survey. 

The  satisfaction  of  the  clientele  of  an  organization  is  certainly 

O 

a major  goal  of  public  agencies.  Indeed,  some  directors  and 
clinicians  argued  that  the  satisfaction  of  the  clientele  should  be  the 
only  goal  of  these  organizations.  Yet,  as  was  pointed  out  earlier, 
these  organizations  receive  most  of  their  funds  from  public  taxes, 

^One  organization  did  construct  a much  more  detailed  consumer 
satisfaction  survey  and  planned  to  use  interviewers  to  carry  it  out. 

g 

See  Aaron  Wildavsky,  The  Politics  of  the  Budgetary  Process 
(Boston:  Little,  Brown,  and  Company,  1964),  p.  66. 
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which  means  that  they  must  be  responsive  to  the  values  of  the  general 
public  and  not  just  the  recipients  of  their  services.  Often  the  goals 
of  the  general  public  and  groups  such  as  the  family  of  the  clientele 
are  in  open  conflict  with  the  goal  of  satisfying  the  patient.  The 
public  is  usually  interested  in  keeping  the  costs  of  these  public 
organizations  as  low  as  possible.  The  family  often  wishes  to  keep 
the  client  away  from  home.  The  following  director  notes  the  in- 
compatibility that  often  occurs  between  satisfying  the  patient  and 
satisfying  his  family  or  the  public: 

(Director)  I find  that  most  of  the  people  who  come  to 
my  office  dissatisfied  with  the  service  are  dissatisfied 
because  they  wanted  the  treatment  to  go  in  one  direction  and 
the  physician  wanted  the  treatment  to  go  in  another.  I think 
that  this  is  especially  true  with  the  family,  who  say,  "Hell, 
this  guy  came  in  and  you  recommended  him  for  outpatient 
treatment.  Hell,  I wanted  him  sent  to  the  State  Hospital. 

Keep  him  locked  up.  I don't  care  what  you  do  with  him.  Don't 
send  him  back  to  us.  " Many  of  the  patients  are  perfectly 
happy  to  be  hospitalized  in  the  inpatient  unit.  It  is  heated, 
lighted,  and  air-conditioned.  It  is  a hell  of  a lot  better  than 
the  homes  of  the  ghetto.  And  the  clients  are  not  particularly 
satisfied  with  the  service  when  they  are  being  discharged.  9 

In  short,  consumer  satisfaction  surveys  focus  upon  one  of  the 

important  goals  of  these  organizations,  but  the  public  holds  other 

goals  for  these  organizations  than  simply  satisfying  their  clientele. 


Q 

In  this  example  the  conflict  between  the  satisfaction  of  the  client 
and  the  goals  of  the  public  concerns  the  desire  of  many  of  the  clients  to 
remain  in  the  expensive  inpatient  unit  which  costs  more  tax  dollars 
to  support  than  the  less  expensive  outpatient  clinic. 
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However,  a more  serious  objection  to  consumer  satisfaction 
surveys  is  that  they  do  not  truly  measure  the  quality  of  service  that 
clients  receive.  Some  directors  and  clinicians  pointed  out  that  a 
large  percentage  of  poor  clients  had  never  had  any  form  of  treatment 
previously  and  were  likely  to  be  satisfied  with  virtually  any  attempt 
to  help  them.  Indeed,  every  consumer  satisfaction  study  examined 
by  this  author  was  highly  positive,  with  70  percent  or  more  of  the 
clientele  of  various  mental  health  centers  judging  the  services  to 
be  at  least  satisfactory  if  not  excellent.  ^ By  way  of  contrast, 
evaluations  which  focus  upon  behavioral  changes  tend  to  be  negative.  ^ 
A cynic  would  argue  that  the  likelihood  of  positive  findings  explains 
the  popularity  of  the  consumer  satisfaction  survey.  Certainly,  the 
uniformity  of  these  highly  positive  results  makes  it  impossible  to 
use  these  surveys  to  distinguish  between  those  organizations  which 
are  doing  a better  than  average  job  and  those  which  are  doing  a 


See,  e.  g.  , Gerald  Landsberg,  "Consumers  Appraise  Store- 
front Mental  Health  Services,  " Evaluation,  Vol.  1,  No.  2 (1973), 
pp.  66-76,  and  James  B.  Goyne  and  Paulette  Ladoux,  "Patients' 
Opinions  of  Outpatient  Clinic  Services,  " Hospital  and  Community 
Psychiatry,  Vol.  24  (September,  1973),  pp.  627-628. 

^For  examples  of  resistance  to  behaviorally-oriented 
evaluations,  see  George  Fairweather,  et  al.  , Community  Life  for 
the  Mentally  111:  An  Alternative  to  Institutional  Care  (Chicago: 
Aldine  Publishing  Company,  1969),  p.  341  and  G.  M.  Carstairs, 
"Problems  of  Evaluative  Research,  " in  Community  Mental  Health: 
An  International  Perspective, ed„  Richard  Williams  and  Lucy  Ozarin 
(San  Francisco:  Jossey  Bass,  1968),  pp.  44-62. 
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mediocre  job.  On  the  other  hand,  such  studies  may  provide  a useful 
check  against  really  poor  centers.  If  a consumer  satisfaction  study 
finds  a low  rate  of  satisfaction,  something  may  be  wrong. 

Despite  its  popularity  among  these  fourteen  organizations, 
there  were  few  examples  of  the  use  of  the  consumer  satisfaction 
survey  for  changing  the  operations  of  the  organizations.  In  a few 
instances,  it  did  provide  feedback  that  aided  in  discovering  problems 
of  the  organization: 

(Evaluator)  One  of  the  things  that  our  evaluation  has 
made  us  aware  of  is  the  fact  that  we  have  to  be  always  con- 
cerned with  a waiting  list  for  our  clinic  because  we  see  our- 
selves as  a crisis  intervention  center.  One  specific  example 
occurred  when  we  had  a psychiatric  consultant  who  came  two 
days  a week  for  a while.  His  primary  role  had  never  been 
defined  clearly.  But  we  became  very  much  aware  through  our 
program  evaluation  project  that  on  the  days  that  he  was  there, 
people  waited  for  long  periods  of  time  in  the  waiting  room.  It 
made  the  people  upset.  We  traced  it  back  and  found  out  that 
he  was  not  just  handling  the  medication  as  we  had  supposed. 

He  was  doing  psychotherapy  and  taking  an  hour  or  more  out 
of  necessity.  We  defined  his  role  more  clearly  as  taking  care 
of  medicine  only.  If  any  need  for  long  term  therapy  arose,  he 
would  channel  the  patients  to  us.  Things  went  much  smoother. 

Other  directors  argue  that  they  can  secure  the  same  information 

without  conducting  a formal  consumer  satisfaction  survey  by  simply 

listening  to  complaints  registered  by  clients.  However,  it  is 

possible  that  there  are  situations  in  which  a director  or  program 

head  finds  such  information  useful  in  convincing  others  in  the 
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organization  to  change  the  nature  of  their  operations.  ^ 

Another  clinician  admitted  that  very  general  questions  were 
asked  in  these  surveys  but  noted  that  sometimes  useful  information 
about  the  overall  flow  of  the  Center's  operations  turned  up: 

(Clinician)  We  made  up  our  own  consumer  satisfaction 
scale  which  was  quite  elementary  with  obvious  questions  such 
as,  "Are  you  happy  with  the  service  you  received?  " We  had 
ten  questions  like  that.  I think  it  was  very  useful.  We  began 
to  look  to  see  what  was  happening  dis positionally  to  the  cases 
and  were  quite  surprised  to  find  that  in  most  of  the  cases, 
people  were  seen  only  two  or  three  times.  It  revamps  some 
of  our  thinking  about  ourselves.  It  is  sort  of  taking  a survey 
of  how  the  system  flows.  It  is  an  evaluation  of  another  sort. 

Again,  it  is  likely  that  alert  therapists  and  administrators  already 

have  an  idea  of  how  many  (or  few)  times  they  see  their  clients,  but 

these  surveys  may  be  useful  in  providing  quantitative  proof  of  such 

. . n 
suspicions. 

A curious  finding  is  that  many  directors  of  the  centers  did 
not  expect  to  get  much  useful  information  from  these  surveys. 


1 2 

For  example,  the  use  of  objective  data  to  reduce  the  mutual 
tension  involved  in  giving  negative  feedback  to  a subordinate  has 
been  pointed  out  by  Peter  M.  Blau,  The  Dynamics  of  Bureaucracy 
(2nd  ed. ; Chicago:  University  of  Chicago  Press,  1963),  p.  43. 

1 3 

However,  clinicians  at  centers  which  had  not  undertaken 
such  surveys  specifically  mentioned  the  fact  that  they  saw  their 
clients,  on  the  average,  only  two  or  three  times.  They  went  on  to 
argue  that  this  limited  contact  seemed  to  be  a good  reason  not  to 
do  a formal  evaluation  because  the  effects  of  the  therapy  were  not 
likely  to  be  great  under  such  conditions. 
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Despite  this  fact,  they  carry  them  out  anyway: 

(Director)  We  are  going  to  do  something  because  we 
feel  we  have  the  capability.  We  just  put  in  a computer  system. 
We  feel  that  we  have  the  capability  of  carrying  out  some  sort 
of  evaluation.  It  [the  consumer  satisfaction  survey]  consists 
of  twenty  questions.  Basically,  we  are  asking  what  patients 
think  about  the  Center.  It  will  go  out  to  all  of  the  new  patients 
seen  in  the  last  year,  about  4,  000.  A second  phase  will  be 
to  go  out  and  do  personal  interviews  with  about  300  randomly 
selected  patients.  With  these  two  instruments,  I think  we 
will  get  a pretty  good  idea  not  whether  we  are  doing  anything, 
but  whether  the  patients  think  we  are  doing  anything. 

One  director  questions  the  ability  of  the  clients  to  give  an  objective 

appraisal  of  the  services  they  receive: 

(Director)  Well,  I will  be  surprised  if  it  shows  me 
something  that  I didn't  already  expect.  Most  of  the  people 
(and  I will  be  surprised  if  we  get  much  more  than  a 15  percent 
response)  probably  will  be  tremendously  angry  or  pleased 
with  the  Center  because  either  it  didn't  perform  all  of  the 
miracles  it  was  supposed  to  perform  or  because  it  was  a 
nice  place  and  people  care.  I think  probably  both  of  these 
groups  will  be  sending  the  questionnaires  in.  . . . 

The  directors  of  centers  that  had  not  carried  out  such  surveys 

often  argue  that  similar  information  can  be  collected  in  less  expensive 

ways: 


(Director)  No,  I don't  think  it  would  be  useful  [i.  e.  . 
undertaking  a consumer  satisfaction  survey] . In  this  particular 
consumer  satisfaction  study  [he  is  referring  to  a consumer 
satisfaction  study  he  had  read  about  in  a journal]  85  percent 
said  they  were  satisfied.  I would  expect  that  if  we  sent  out 
. a similar  questionnaire,  that  by  and  large,  we  would  get 
similar  results.  Because  most  people  are  satisfied  with  the 
treatment.  The  reason  why  I say  that  is  if  they  weren't 
satisfied  with  the  treatment,  I think  that  the  population  is 
vocal  enough  to  let  it  be  known  that  they  are  not.  I can 
personally  say  that  because  this  office  is  where  a lot  of  the 
complaints  come. 


42 


Consumer  satisfaction  surveys  were  not  perceived  by  the 
center  staff  as  threatening.  Using  such  surveys  to  introduce  a 
program  of  research  and  evaluation  to  the  staff  is  one  of  the  positive 
examples  concerning  their  impact: 

(Clinician)  The  first  thing  that  it  [ consumer  satisfaction 
survey]  accomplished  was  that  it  introduced  the  staff  generally 
to  the  concept  of  evaluation.  I think  that  everyone  knows  that 
that  is  the  first  big  step. 

However,  in  summary,  there  were  only  a few,  isolated 
examples  of  cases  where  the  consumer  satisfaction  surveys  had 
influenced  decisions  concerning  the  delivery  of  services.  Even  more 
significantly,  the  surveys  are  not  generally  expected  to  have  much 
impact  on  the  decision  making  of  these  organizations.  14  Nevertheless, 
significant  amounts  of  time  and  money  are  being  devoted  to  carrying 
them  out. 


Goal  Attainment  Scaling 

The  goal  attainment  scaling  method  of  evaluation  embraces 
a wide  range  of  specific  methods  including  the  following  major 
variations: 


(1)  Original  Method:  As  originally  developed  at  the 
Hennepin  Mental  Health  Center  in  Minneapolis,  Minnesota, 
goals  are  set  for  the  client  by  an  intake  worker.  Then  the 
client  is  assigned  to  a particular  form  of  treatment.  After 


4Here,  the  author  is  referring  to  the  internal  decision  making 
of  the  organizations.  In  later  chapters,  other  types  of  functions  of 
evaluation  will  be  examined,  such  as  use  in  influencing  funding  agencies 
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a specified  length  of  time,  a follow-up  worker  interviews  the 
patient  in  order  to  ascertain  to  what  extent  the  goals  set  at 
intake  have  been  fulfilled. 

(2)  Contract  Fulfillment  Analysis  Methods:  With  this 
method,  goals  are  set  by  negotiation  between  the  therapist 
and  the  client.  However,  the  degree  of  goal  attainment  is 
determined  as  in  the  above  method,  with  a follow-up  worker 
interviewing  the  client  to  see  to  what  degree  goals  have  been 
attained.  1 5 

(3)  Standardized  Methods:  In  these  forms,  there  are 
sets  of  prespecified  goals  categorized  according  to  problem 
areas.  Due  to  its  standardization,  it  has  been  possible  to 
computerize  this  form,  in  contrast  to  the  above  forms  in 
which  everything  is  done  by  hand. 

Elements  common  to  all  forms  of  the  goal  attainment  model 
of  evaluation  include  the  following:  (1)  a six-point  scale  is  used  to 
determine  the  degree  to  which  a goal  has  been  attained,  ranging 
from  the  "most  unfavorable  success"  to  the  "best  anticipated 
success";  (2)  the  goals  are  weighted  according  to  their  degree  of 
importance  by  the  person  (or  persons)  who  sets  the  goals;  (3)  after 
therapy  has  been  completed,  a single  goal  attainment  score  is  com- 
puted based  on  the  degree  of  goal  attainment  for  each  individual  goal 
set  and  its  "weighted  importance.  "17 


15This  description  is  based  on  Collier  County  Mental  Health 
Clinic,  "Progress  Report  on  Program  Evaluation  Project,  " Naples, 
Florida,  n.  d.  (Mimeographed.  ) 

1 6 

See  Nancy  C.  Wilson,  "The  Tri-Informant  Goal -Oriented 
Progress  Note"  (Paper  presented  at  the  American  Psychological 
Association,  Honolulu,  Hawaii,  1972). 

1 7 

See  Thomas  J.  Kiresuk  and  Robert  Sherman,  "Goal  Attain- 
ment Scaling:  A General  Method  for  Evaluating  Comprehensive  Com- 
munity Mental  Health  Programs,  " Community  Mental  Health  Journal, 
Vol.  4 (December,  1968),  pp.  443-453. 
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The  major  differences  among  the  many  versions  of  goal 

attainment  evaluation  concern  mainly  the  issues  of  who  sets  the 

goals  and  what  types  of  goals  are  to  be  included.  In  some  forms, 

only  the  therapist  is  involved  in  the  setting  of  goals.  In  others,  not 

only  the  client  but  family  members  or  others  become  involved  in 
1 8 

the  goal-setting. 

The  originators  of  the  goal  attainment  method  of  evaluation 
put  a great  deal  of  emphasis  upon  having  behavioral  indicators  for 
each  of  the  more  generalized  goals  set.  Thus,  for  the  treatment 
of  drinking  problems,  specific  amounts  of  alcoholic  intake  might  be 
set  as  the  means  by  which  the  level  of  attainment  is  measured. 

Recently,  there  has  been  a movement  towards  the  inclusion  of 
concepts  for  which  it  is  difficult  if  not  impossible  to  use  this  kind 
of  measure,  such  as  "self-concept"  goals. 

Finally,  there  is  a distinction  between  those  centers  in  which 
an  individualized  goal  is  constructed  anew  for  each  client  and  other 
centers  in  which  lists  of  pre-set  goals  are  used.  These  lists  include 
both  behavioral  and  non-behavioral  goals.  Some  examples  are 
given  below: 

^%ee  Nancy  C.  Wilson,  "The  Tri-Informant  Goal-Oriented  Progress 
Note.  " 


45 


Specific  Goals 

(1)  Gain  or  lose  weight,  reduce 
or  stop  drinking,  improve 
nutrition 

(2)  Motor  Behavior  (e.  g. , 
posturing),  sexual  behavior 
(e.  g. , exhibitionism),  activity 
disorders  (e.  g.  , hyperactivity) 

(3)  Self-regard,  self-trust, 
self-confidence  in  various  abilities, 
changing  independence, 
appropriate  handling  of  feelings.  ^ 

The  standardized  form  includes  items  to  indicate  the  level  of  success 

that  is  expected  to  be  attained  and  the  degree  of  importance  attached 

to  each  goal. 

Taken  together,  the  various  forms  of  goal  attainment  evaluation 

represent  the  single  fastest  growing  type  of  evaluation  in  community 

mental  health  centers  and  are  also  achieving  a rapid  diffusion  to 

20 

fields  other  than  mental  health.  In  addition,  the  reliability  of 
the  goal  attainment  scores  has  been  tested  and  found  to  be  fairly 
high.21 


Category  of  Goals 

(1)  Medical  Goals 

(2)  Symptom  Goals 

(3)  Self-Concept  Goals 


1 Qmi 

These  examples  are  taken  from  a form  developed  by  Nancy 
C.  Wilson  of  the  Fort  Logan  Mental  Health  Center,  Denver,  Colorado. 

20 

See  Howard  Davis,  "Four  Ways  to  Goal  Attainment:  An 
Overview,  " Evaluation,  Vol.  1,  No.  2 (1973),  pp.  43-48. 

9 1 

1See  Geoffrey  Garwick,  "First  Reliability  Report  for  Goal 
Attainment  Scaling,  " P.  E.  P.  Newsletter  Compendium  (Minneapolis: 
Program  Evaluation  Project,  n.  d. ),  p.  54. 
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The  scope  of  the  goal  attainment  method  is  impressive  because 
this  same  technique  can  be  applied  to  all  types  of  programs,  both 
direct  and  indirect.  Although  the  goal  attainment  score  is  calculated 
for  the  success  of  the  service  delivered  by  an  individual  clinician, 
the  scores  of  several  clinicians  can  be  averaged  together  to  give 
a goal  attainment  score  for  an  entire  program.  23  The  goal  attain- 
ment method  has  even  been  applied  to  the  evaluation  of  the  administra- 
tion of  a mental  health  organization.  24  One  potential  limitation 
of  the  scope  of  this  evaluation  technique  involves  those  forms  which 
require  the  active  cooperation  of  the  client  in  ascertaining  the  goals 
to  be  set  for  him.  This  cooperation  may  be  impossible  to  achieve 
when  the  patient  is  in  a highly  agitated  state.  25  A more  important 
limitation  of  goal  attainment  scaling  is  that  the  model  has  not  yet 
been  related  to  the  budget  and  other  important  areas  of  organizational 


22 

Indirect  services  refer  to  programs  such  as  consultation 
and  education.  In  these  programs,  the  "client"  is  not  experiencing 
difficulty  but  is  taught  how  to  better  handle  people  who  do  experience 
psychological  problems.  These  programs  are  particularly  aimed 
at  groups  who  come  into  frequent  contact  with  individuals,  such  as 
clergymen,  policemen,  and  teachers. 

23 

However,  this  form  of  evaluation  does  not  enable  clear 
distinctions  to  be  made  between  the  effectiveness  of  different  programs. 
Its  focus  remains  the  individual  interaction  between  therapist  and  client. 

See  Sander  Lund,  Crisis  Intervention  Center:  Administrative 
Evaluation,  P.  P,  Newsletter  Compendium  (Minneapolis: 

Program  Evaluation  Project,  n.  d.  ),  p.  86. 

25™. 

This  is  often  the  case  for  clients  admitted  to  the  inpatient 
unit  of  a mental  health  center. 
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decision  making.* 2® 

The  focus  of  goal  attainment  evaluation  is  much  sharper  than 
that  of  consumer  satisfaction  surveys.  Specific  goals  are  set  in 
contrast  to  the  very  generalized  questions  which  are  characteristic 
of  consumer  satisfaction  surveys.  However,  despite  the  satisfactory 
results  from  testing  the  reliability  of  the  goal  attainment  scores,  the 
validity  and  interpretation  of  the  scores  is  still  open  to  question. 

There  is  the  definite  problem  of  bias  in  the  report  of  the 
degree  of  goal  attainment  because  these  scores  are  determined  in 
large  degree  by  the  report  of  the  client.  The  former  client  may 
perceive  the  failure  to  achieve  the  goals  of  the  therapy  as  reflecting 
upon  himself  rather  than  the  therapist.  ^ Such  a client  would  tend 
to  over-report  the  success  of  the  therapy. 

Another  problem  concerns  the  interpretation  of  the  scores. 

A large  percentage  of  the  clients  of  many  of  these  centers  receive 
drugs  as  part  of  their  treatment.  One  director  points  out  that  in 
such  cases  it  is  impossible  to  determine  whether  the  therapy  or 
the  drugs  is  responsible  for  any  improvement  that  may  occur  in 

26Recently,  however,  attempts  have  been  made  to  do  this. 

See  Joseph  Halpern  and  Paul  R.  Binner,  "A  Model  for  an  Output 
Value  Analysis  of  Mental  Health  Programs,  " Administration  in 
Mental  Health  (Winter,  1972),  pp.  40-51. 

2 7 

Indeed,  most  clinicians  agree  that  the  client  plays  a big 
role  in  determining  the  success  of  the  therapy.  This  point  will 
be  discussed  further  in  the  next  chapter. 
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the  client's  condition.  ^ 

Bias  can  also  result  from  the  participation  of  the  therapists 
and  follow-up  workers  in  the  scoring  of  the  degree  of  success.  There 
is  substantial  evidence  that  factors  other  than  "objective  improve- 
ment are  likely  to  affect  this  rating.  ^ Even  when  behavioral 
indices  are  constructed  to  determine  the  degree  of  success,  the 
scores  are  still  based  on  reports  and  perceptions  of  change  rather 
than  actual  observation  of  these  changes  under  controlled  conditions. 

Some  clinicians  and  evaluators  think  that  the  validity  of  goal 
attainment  data  is  even  more  suspect  when  "non -behavioral"  goals 
such  as  self-concept  are  included: 

(Clinician)  Say  that  I ask  you,  "What  do  you  want  to  do?  " 
And  you  say,  "I  want  to  become  a better  person.  " And  then, 
after  I have  completed  the  therapy,  I ask  you,  "Do  you  feel 
like  a better  person?"  You  say,  "Yes.  " And  I say,  "Goal 
Accomplished.  " And  what  the  hell  has  happened?  I would 
prefer  something  more  behaviorally -oriented  but  it  wouldn't 
sell  right  here,  at  this  center.  . . . 

(Evaluator)  Yes,  I don't  see  how  that  can  be  done 
[i.  e. , use  the  goal  attainment  method  for  non -behavioral 
goals],  I personally  favor  the  behavioral  definition.  . . . 

The  Center  uses  a referral  form  and  the  teachers  check 
off  the  items  that  are  on  the  form.  • I was  perturbed  when  I 


2 8 

Approximately  70  percent  of  the  clientele  were  administered 
drugs  in  the  organization  in  which  the  author  was  an  observer. 

^®See,  e.  g.  , Jim  Mintz,  "What  is  'Success'  in  Psychotherapy?" 
Journal  of  Abnormal  Psychology,  Vol.  80  (February,  1972), 

pp.  11-19. 
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saw  the  form  as  it  has  been  used  in  the  past  because  there 
are  things  like  poor  self-concept  on  it.  Now,  how  the  hell 
is  the  teacher  going  to  know  what  the  kid's  self-concept  is? 

I think  even  the  psychiatrist  would  have  quite  a bit  of  trouble 
getting  at  that  self-concept.  You  know  that  I personally 
favor  definitions  that  are  observable. 

Due  to  its  sharper  focus,  the  goal  attainment  method  of 

evaluation  appears  to  be  more  likely  to  be  used  by  therapists.  This 

feedback  concerning  the  degree  of  success  which  he  has  achieved 

with  various  clients  may  be  quite  useful  for  the  therapist  and  enable 

him  to  improve  his  effectiveness.  However,  there  are  no  data  to 

back  up  this  contention.  Goal  attainment  scaling  was  implemented 

in  a completely  non-threatening  way  in  which  careful  attention  was 

given  to  keeping  "negative  feedback"  confidential.  Two  different 

evaluators  described  at  length  the  care  that  was  given  to  avoid 

threatening  therapists  with  such  information: 

(Evaluator  #1)  I can  tell  you  what  we  did.  We  have 
made  it  extremely  confidential.  We  discussed  what  to  do 
with  negative  feedback  and  decided  that  the  only  one  to  see 
the  results  would  be  the  follow-up  worker.  We  decided  the 
thing  was  confidential.  It  ended  up  as  a joke  that  we  were 
getting  our  report  cards.  So  the  follow-up  worker  was  the 
only  one  who  knew  the  results  and  tallied  them  up  and  put 
them  in  an  envelope.  We  decided. to  provide  all  of  the  positive 
comments  but  only  negative  ones  if  there  were  some  consistency 
and  left  it  up  to  the  judgment  of  the  follow-up  worker. 

(Evaluator  #2)  Well,  you  would  go  through  the  proper 
channels  with  a negative  evaluation.  It  would  be  unthinkable 
to  come  up  with  something  like  this  without  airing  it  out. 

For  one  thing,  you  would  lose  the  trust  and  confidence  of  the 
people,  and  just  hurt  yourself.  If  the  Center  director  wants 
to  make  an  issue  of  it,  that  is  fine.  The  point  is  that  I 
would  present  a statement  to  the  Center  director  including 
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my  feelings  as  to  what  should  be  done.  I wouldn't  say,  "Do 
this  or  I will  quit.  " 

The  only  evidence  of  substantive  impact  of  goal  attainment 
scaling  is  a generalized  feeling  on  the  part  of  those  therapists  and 
directors  committed  to  it  that  it  more  clearly  defines  the  purposes 
of  the  therapy: 

(Director)  Yes,  I think  the  unit  which  had  introduced 
goal  attainment  scaling  is  much  more  comfortable  in  getting 
a handle  on  whether  their  clients  are  getting  any  help  or  not. 

I know  that  they  use  that  as  a way  to  structure  the  program 
for  individual  clients.  They  seem  pretty  well  satisfied  with  it. 

(Clinician)  One  is  that  goal  attainment  scaling  in  each 
encounter  gets  the  client  involved  in  the  setting  of  goals. 

Many  patients  find  that  really  usetul.  It  defines  whv  thev  are 
coming.  It  also  gives  them  a stepladder  on  which  to  decide. 
Did  I or  did  I not  achieve  it?  " For  example,  one  of  the  goals 
might  be  to  get  a job  so  the  expected  level  might  be  that  in  the 
next  two  weeks  the  person  has  made  an  attempt  to  get  a job. 
The  highest  level  might  be  that  he  has  made  twenty  attempts 
to  get  a job.  Most  of  the  time  people  come  in  and  complain 
about  behavior.  They  don't  like  what  their  boss  is  aoing  or 
what  they  themselves  are  doing. 

There  are  no  "hard  data"  to  indicate  that  the  effectiveness  of  the 

therapy  increases  after  the  introduction  of  goal  attainment  scaling. 

Also,  other  than  being  used  to  provide  confidential  feedback  to 

individual  therapists,  it  does  not  appear  that  goal  attainment 

evaluation  has  any  highly  visible  impact  on  the  operations  of  these 


organi  zations . 

The  sharper  focus  of  goal  attainment  scaling  creates  the 
potential  for  conflict  between  different  philosophies  and  methods  of 
treatment.  One  director  noted  that  the  method  seems  to  be  helpful 


in  one  program  but  that  he  would  hesitate  to  introduce  it  in  other 
programs : 
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(Director)  It  was  easy  for  the  one  particular  service  to 
implement  because  they  only  had  a staff  of  six  people.  The 
director  of  the  service  is  behaviorally-oriented  and  had  a 
great  deal  of  personal  faith  and  belief  in  the  need  for  this. 

He  communicated  this  to  his  staff.  Then  the  staff  adopted  the 
attitude  of  the  director.  But  with  a variety  of  programs  and 
directors  of  various  faiths --it  is  more  difficult. 

A psychodynamically -oriented  clinical  psychologist  indicates  his 

opposition  to  the  general  concept  of  setting  specific  goals  for  the 

therapy: 


(Clinician)  Yes,  if  you  are  going  to  look  at  treatment  in 
those  kinds  of  behavioral  terms,  I think  this  is  where  my 
approach  and  [name  of  another  therapist]  differ.  He  very 
clearly  sets  up  goals  and  motivation  whereas  I don't  presume 
motivation.  I don't  give  the  patient  credit  for  being  able  to 
verbalize  motivation  or  goals.  There  are  probably  a lot  of 
things  he  doesn't  know  he  is  doing,  ^wants  unconscious. 
Sometimes  I have  no  specific  goal. 

Except  when  goals  are  standardized,  it  is  difficult  to  make 
comparisons  between  different  therapists  with  the  goal  attainment 
method  of  evaluation.  Therapists  with  lower  goal  attainment  scores 
can  claim  that  they  simply  are  setting  more  ambitious  goals  than 
the  others.  Standardization  of  goals  is  likely  to  facilitate  the  use 


30Nearly  every  form  of  evaluation  presumes  a goal.  However, 
note  what  Michael  Scriven  has  called  "goal-free  evaluation.  " See 
Michael  Scriven,  "Prose  and  Cons  About  Goal-Free  Evaluation,  " 
Evaluation  Comment,  Vol.  3 (December,  1972),  pp.  1-4. 


of  this  form  of  evaluation  for  use  in  making  decisions,  as  the 
following  director  argues: 
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(Director)  My  own  personal  bias  is  towards  something 
that  has  a little  more  standardization.  I guess  I am  a little 
more  systems -oriented.  I like  to  be  able  to  make  some  kinds 
of  comparison  which  I don't  think  that  you  can  do  very  well 
with  goal  attainment  scaling  [ i.  e.  , the  unstandardized  forms]. 
So,  I would  probably  lean  toward  something  that  is  a little 
more  standardized. 

However,  attempts  to  standardize  goals  also  are  likely  to  run  into 
opposition.  The  following  director  describes  the  reasons  for 
opposition  to  the  goal  attainment  method  of  evaluation  and  particularly 
to  the  more  standardized  forms: 

(Director)  I think  it  [goal  attainment  scaling]  sounds 
good,  but  in  therapy  it  is  not  that  simplistic.  Our  goals 
keep  changing.  I think  that  there  is  a value  in  that --in  having 
a leeway  in  the  kind  of  goal  that  you  are  trying  to  obtain.  I 
would  hate  to  see  the  day  when  NIMH  tells  us  that  we  had  to 
have  this  kind  of  evaluation  because  I don't  feel  that  we  have 
reached  the  state  where  we  can  set  up  definite  goals  or  that 
they  should  depend  on  the  patient. 

In  short,  goal  attainment  evaluation  is  likely  to  have  a positive 

impact  on  therapy  only  when  those  using  this  form  of  evaluation 

agree  with  its  purposes  and  feel  that  it  would  be  useful.  Otherwise, 

it  is  likely  to  have  a negative  impact  on  the  work  of  the  therapists. 

To  summarize  the  impact  of  goal  attainment  scaling,  it  has 

been  used  mainly  to  provide  feedback  to  the  therapists  concerning 

the  results  of  their  therapy.  Some  clinicians  and  directors  felt 

that  it  has  enabled  both  therapists  and  clients  to  get  a "better  handle" 

on  the  purpose  of  the  therapy.  However,  there  was  no  systematic 
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attempt  to  test  the  assumption  that  it  has  led  to  greater  effectiveness 
of  the  therapists. 

Goal  attainment  scaling  evaluations,  like  consumer  satisfaction 
surveys,  consume  significant  resources  of  the  organizational  staffs. 
If  taken  seriously,  the  goal  attainment  forms  require  a large  amount 
of  the  therapist’s  time.  ^ A follow-up  worker  is  also  needed  in 
order  to  measure  the  degree  of  attainment. 

However,  despite  the  resources  of  time  and  staff  being  devoted 
to  this  form  of  evaluation,  it  was  not  actively  used  in  making  the 
major  decisions  of  the  organization,  such  as  those  related  to  the 
budget.  It  generally  was  introduced  in  such  a non-threatening  way 
that  it  was  also  not  used  to  make  major  personnel  decisions  either.  ^ 


Epidemiological  Surveys 

Psychiatric  epidemiological  surveys  attempt  to  measure  the 

incidence,  prevalence,  and  distribution  of  mental  illness  in  a 
33 

particular  area.  Such  surveys  can  be  used  as  evaluative  tools 


31See  Gregg  Wahlstrom,  "A  Clinician's  Reaction  to  Goal  Attain- 
ment Scaling,  P.  E.  P.  Newsletter  Compendium  (Minneapolis: 
Program  Evaluation  Project,  n.  d.  ),  pp.  43-44. 

32 

The  reasons  for  the  non -threatening  way  that  these  evaluations 
■were  used  are  explored  in  following  chapters. 

33 

These  surveys  are  also  called  "needs  assessment  surveys.  " 
However,  the  latter  term  is  also  used  to  refer  to  surveys  which 
measure  general  needs  and  do  not  use  psychiatric  definitions  of 
mental  illness  or  the  methodology  of  survey  research. 
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in  at  least  two  ways:  (1)  to  compare  the  populations  which  the 
epidemiological  surveys  show  to  be  in  the  worst  state  of  health 
with  those  being  served  by  the  mental  health  centers  in  order  to 
see  if  the  center  is  reaching  those  most  in  need;  (2)  to  repeat  the 
survey  after  a given  period  of  time  in  order  to  see  if  the  organization 
has  been  successful  in  decreasing  the  occurrence  of  mental  illness. 

Psychiatric  epidemiological  surveys  are  not  designed  to 
provide  feedback  on  the  quality  of  the  programs  or  therapists  of 
mental  health  organizations.  In  certain  cases,  a program  may  have 
the  responsibility  of  treating  a particular  group  of  persons  who  can 
be  identified  by  the  surveys  such  as  a particular  problem  group 
(e.  g.  , alcoholics)  or  age  group  (e.  g.  , children).  In  such  cases,  if 
the  prevalence  and  incidence  of  mental  illness  change  markedly 
for  the  better  or  worse  in  comparison  with  the  rest  of  the  population, 
the  results  of  the  epidemiological  surveys  may  actually  have  fairly 
clear  implications  for  individual  programs.  3^  Generally,  however, 
these  surveys  are  aimed  at  testing  certain  relationships  that  do 
not  have  any  direct  utility  for  individual  therapists  or  programs.  35 

3 4 

In  one  case,  certain  practitioners  were  allowed  to  put 
items  into  the  survey  that  they  felt  might  provide  useful  information 
for  the  programs  of  which  they  were  members. 

35 

For  evidence  supporting  this  statement,  see  below. 
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If  everyone  were  agreed  as  to  the  definition  of  mental  illness 
and  health,  and  how  they  could  be  measured,  then  it  could  be  strongly 
argued  that  epidemiological  studies  approach  as  closely  as  is 
possible  a truly  accomplishment -oriented  evaluation.  However,  no 
such  consensus  has  yet  been  reached  concerning  how  to  define  mental 

q o 

illness  or  health.  Moreover,  epidemiological  surveys  have  produced 

great  ranges  in  the  extent  of  psychiatric  "impairment"  found  in  the 

populations  studied,  ranging  from  a few  percent  to  more  than  50 
37 

percent.  It  is  apparent  that  different  operational  definitions  of 
mental  illness  can  lead  to  large  differences  in  the  amount  of  mental 
illness  found  and  the  conclusions  reached  from  such  a survey. 

Another  problem  with  epidemiological  surveys  is  that  they 
violate  the  concept  of  community  mental  health  by  attempting  to 
apply  a.  single  standard  of  mental  health  to  all  communities,  re- 
gardless of  the  particular  value  systems  of  these  communities. 

Indeed,  several  clinicians,  particularly  those  in  the  rural  areas, 
noted  the  fact  that  the  communities  they  served  tolerated  individuals 
which  other  communities  would  regard  as  extremely  deviant.  There 

q q 

For  a summary  of  several  different  approaches  to  the  concept 
of  mental  illness,  see  S.  B.  Sells,  ed.  , The  Definition  and  Measure- 
ment of  Mental  Health  (Washington:  Government  Printing  Office,  19  68). 

i See  Bruce  P.  Dohrenwend  and  Barbara  Snell  Dohrenwend, 

The  Problem  of  Validity  in  Field  Studies  of  Psychological  Disorder,  " 
Journal  of  Abnormal  Psychology,  Vol.  70  (February,  1965),  pp.  52-69. 


56 


are  current  efforts  to  develop  more  "neutral"  definitions  of  mental 
illness.  Until  a greater  consensus  is  reached  on  both  conceptual 
and  operational  definitions  of  mental  illness,  debates  concerning 
the  validity  and  utility  of  these  surveys  are  likely  to  continue. 

The  most  serious  problem  in  using  epidemiological  surveys 
as  evaluative  tools  is  the  lack  of  clarity  concerning  their  meaning. 
Many  variables  other  than  the  quality  or  quantity  of  mental  health 
services  provided  by  mental  health  organizations  affect  the  degree 
of  mental  illness  found  by  epidemiological  surveys.  Some  of  these 
extraneous  factors  are  population  shifts,  socioeconomic  conditions, 
general  cultural  trends,  and  the  impact  of  other  public  agencies. 

At  one  Center  where  the  author  was  an  observer,  the  evaluator 
asked  a psychiatrist  how  he  thought  the  quality  of'  the  services  of 
the  Center  should  be  evaluated.  The  psychiatrist  replied  that  the 
only  way  to  do  it"  was  to  carry  out  an  epidemiological  study  and 
repeat  it  after  a certain  interval  in  order  to  see  to  what  extent  the 
figures  on  prevalence  had  changed.  Later,  however,  in  a discussion 
involving  the  psychiatrist,  the  evaluator,  and  a consultant  (who  was 

38 

For  an  illuminating  account  of  the  differences  between 
providing  services  in  a rural  setting  and  an  urban  setting,  see  Hans 
R.  Huessy,  Tactics  and  Targets  in  the  Rural  Setting,  1 in  Handbook 
of  Community  Mental  Health,  ed.  Stuart  E.  Golann  and  Carl  Eisdorfer 
(New  York:  Apple  ton -Century -Crofts,  1972),  pp.  699-710.  Huessy 
concludes  that  practices  that  are  regarded  as  good  technique  else- 
where can  lead  to  damaging  results  in  rural  areas. 
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an  expert  on  needs  assessment  surveys),  even  the  psychiatrist 
admitted  that  the  epidemiological  survey  would  provide  ambiguous 
data: 


(Consultant)  Can  you  use  an  epidemiological  instrument 
to  evaluate  the  impact  of  programs  ? I am  not  entirely 
convinced  that  you  can. 

(Evaluator)  It  is  very  difficult  to  tell  whether  or  not 
the  change  was  the  result  of  the  Center  or  other  factors 

(Psychiatrist)  I still  think  that  it  is  a good  instrument. 
Someway  or  another,  we  have  to  decide  whether  or  not  we  are 
doing  a good  job. 

(Consultant)  But,  take,  for  example,  a case  in  which 
several  DWI's  [persons  arrested  for  driving  while  under  the 
influence  of  alcohol]  are  arrested  here  while  passing  through 
this  county.  That  goes  into  the  records  as  increasing  the 
amount  of  alcoholism  in  this  country.  . . . 

(Psychiatrist)  Or  you  could  argue  that  it  might  have  been 
worse  if  we  weren't  in  operation.  That  is  what  I was  prepared 
to  do  if  the  results  turned  out  bad. 

Thus,  it  is  difficult,  if  not  impossible,  to  draw  conclusions  about 

the  quality  of  the  services  of  mental  health  organizations  based  on 

the  results  of  eipdemiological  surveys. 

In  one  Center,  the  epidemiological  survey  was  used  to  determine 

the  proper  location  of  a mental  health  service  so  that  it  would  be 

close  to  the  most  highly  impaired  area.  Another  Center  used  one 

such  survey  to  locate  their  outpatient  clinic.  The  survey  found 

that  nearly  everyone  went  to  the  same  general  practitioner  in  this 
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catchment  area.  According  to  the  personnel  of  this  Center,  this 
doctor  had  been  prescribing  medicine  without  knowing  what  kinds 
should  be  administered  for  psychological  problems.  When  the  out- 
patient clinic  was  located  near  his  office,  he  agreed  to  refer 
patients  with  psychological  problems  to  this  clinic.  Thus, 
epidemiological  surveys  appear  to  be  of  use  in  making  decisions  of 
where  to  locate  services. 

Otherwise,  there  was  little  evidence  that  epidemiological 
surveys  influenced  the  internal  decisions  of  these  organizations. 

As  was  the  case  with  consumer  satisfaction  studies,  many  of  the 
directors  of  the  centers  undertaking  these  epidemiological  surveys 
argued  that  they  could  anticipate  the  findings.  The  author  asked  one 
director  why  he  had  undertaken  these  surveys  if  he  already  knew 
what  the  results  would  be: 

(Director)  As  I indicated  before,  they  are  necessary 
because  you  can't  take  things  for  granted.  For  example, 
in  one  area  we  never  had  any  patients,  but  the  minute  we 
-opened  the  clinic  we  were  overflowing  with  patients.  The 
other  reason  is  that  it  is  important  in  selling  the  programs 
to  the  community  in  order  to  get  community  funds.  You 
can't  go  to  them  without  scientific  data.  You  won't  get  it 
then.  Today  you  have  to  have  scientific  data.  They  will  say. 
Now  let  me  have  the  data  that  show  how  many  and  where  the 
people  are  who  need  these  services.  " So  it  is  an  excellent 
tool  for  helping  to  obtain  funds. 


39"Catchment  area"  refers  to  the  area  and  population  which  these 
organizations  are  responsible  for  serving. 
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As  the  above  interview  demonstrates,  the  purpose  of  the  epidemio- 
logical surveys  seem  more  directed  to  justifying  increases  in  the 
funding  of  the  organizations  than  to  using  them  internally.  40 

Psychiatric  epidemiological  surveys,  like  the  rest  of  the 
evaluation,  involve  significant  costs  of  time  and  money.  In  fact, 
where  these  surveys  were  being  carried  out,  they  were  consuming 
more  time  than  any  other  single  research  activity.  41  Yet,  the  feed- 
back from  such  surveys  is  not  obtained  until  months  or  even  years 
after  the  beginning  of  the  survey.  Also,  the  information  obtained 
from  them  is  far  from  being  clear  or  useful  for  the  making  of  most 
organizational  decisions.  These  disadvantages  led  one  evaluator  to 
admit  that  she  would  prefer  to  work  on  something  that  the  clinical 
staff  would  find  to  be  more  useful: 

(T valuator ) Yes,  the  epidemiological  survey  was  my 
main  purpose  for  coming  down  here  because  I had  already 
worked  with  the  instrument.  But  the  things  that  I am  now 
interested  in,  the  research  that  really  turns  me  on,  are  those 
^things  which  I can  see  will  be  useful  in  the  organization  such 
as  a follow-up  study  on  people  who  called  up  for  an  appointment 
and  are  not  coming  back  to  keep  the  appointments  for  some 
reason.  This  is  something  that  could  lead  to  actual  changes 
and  affect  the  patient. 


^This  point  is  discussed  further  in  later  chapters. 

41 

For  an  argument  that  much  less  expensive  and  time-consuming 
epidemiological  surveys  can  be  carried  out,  see  R.  David  Mustian  and 
Joel  J.  See,  "Indicators  of  Mental  Health  Needs:  An  Empirical  and 
Pragmatic  Evaluation,  " Journal  of  Health  and  Social  Behavior,  Vol. 

14  (March,  1973),  pp.  23-27. 
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Clinicians,  while  admitting  that  the  information  would  be 
interesting  when  collected,  found  that  they  had  much  more  pressing 
demands  for  direct  services.  These  demands  were  making  it 
impossible  for  them  to  participate  in  carrying  out  the  needs 
assessment  survey: 

(Supervisory  Clinician)  I approached  the  evaluation 
staff  to  do  a follow-up  study  for  patients  seen  in  the  clinic 
in  order  to  get  an  overall  view  of  the  community  attitudes 
towards  the  patient,  and  also  to  get  some  impressions  from 
other  agencies,  but  we  haven't  done  this  because  we  are  in 
the  midst  of  a needs  assessment  study.  I think  that  the  needs 
assessment  study  is  very  valuable.  We  don't  really  know  the 
extent  of  mental  illness,  of  alcoholism,  and  where  the  people 
go  for  service.  But  it  is  going  to  take  a lot  more  effort  than 
we  are  putting  into  it  right  now.  It  is  difficult  to  have  my 
staff  spend  four  or  five  hours  a week  on  it  as  they  are  now  and. 
cut.  . . our  other  programs.  We  have  figured  out  that  it 
would  take  three  years  to  complete  the  study.  In  three  years, 
well,  the  results  would  be  invalid- -if  you  spread  it  over  that 
length  of  time.  I don't  know  what  is  going  to  happen  unless 
they  have  the  money  to  hire  interviewers  to  go  out  and  do  some 
interviews. 

In  short,  when  the  staff  sees  significant  amounts  of  resources 
being  spent  on  an  activity  which  it  sees  having  little  substantive 
impact  on  the  organization,  it  is  likely  to  question  its  value. 


Problem -Oriented  Records 

The  fourth  kind  of  scientific  evaluation  used  is  problem - 
oriented  records.  The  idea  behind  problem-oriented  records 
closely  parallels  the  concept  of  program  budgeting.  It  sets  up  a 
record  system  organized  around  the  problems  of  the  patient  rather 
than  the  usual  system  in  which  the  categories  are  not  systematically  related 


61 


to  the  problems  of  the  client.  42 

As  an  evaluative  tool,  problem-oriented  records  make  it 

possible  for  there  to  be  a better  auditing  of  the  patient's  treatment 
4 3 

and  progress.  Specific  plans  and  goals  are  indicated  in  the 
records  for  each  client.  More  specifically,  a problem-oriented 


record  system  normally  consists  of  the  following  major  components: 


(1)  Data  Base:  This  base  includes  a history  of  the  client, 
his  present  illnesses,  and  the  results  of  any  examination  and 
tests  that  have  been  carried  out  on  him. 

(2)  Problem -List:  This  list  can  include  both  the  complaints 
of  the  patient,  the  analysis  of  the  therapist,  and  the  problems 
indicated  by  the  tests  performed  on  the  patient. 

(3)  Plans  and  Goals : A specific  plan  and  goal  is  associated 
with  each  problem  in  the  problem  list. 

(4)  h oliow  -up : I he  follow-up  includes  progress  notes  and 
further  feedback  concerning  old  problems  or  new  problems.  44 

Ideally,  each  of  these  four  components  is  carried  out  in  sequence 

with  the  feedback  from  the  follow-up  being  used  to  start  the  whole 

sequence  over  again  with  new  data,  new  problems,  and  so  forth. 


42  rpU. 

This  is  analogous  to  the  concept  of  program  budgeting  which 
attempts  to  get  away  from  the  type  of  budgeting  which  is  divorced  from 
the  functional  activities  performed  by  public  organizations.  See 
Leonard  Merewitz  and  Stephen  Sosnick,  The  Budget's  New  Clothes 
(Chicago:  Markham  Publishing  Company,  1971). 

43 

See  Robert  W.  Putsch  etal.  , "Quality  Care,  Problem 
Orientation,  and  the  Medical  Audit,  ' in  The  Problem-Oriented  System, 
ed.  J.  Willis  Hurst  and  H.  Kenneth  Walker  (New  York:  Medcom 
Series,  1972),  pp.  173-182. 

44 

This  description  is  based  on  Harold  Cross,  "The  Problem- 
Oriented  System  in  Private  Practice  in  a Small  Town,  " in  Hurst  and 
Walker,  ibid.  , p.  157. 
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The  scope  of  the  problem-oriented  record  system  is  broad 
because  it  can  be  applied  to  all  interventions  for  which  specific 
problems  and  goals  can  be  identified.  Quantitative  goals  may  be 
made  for  the  client  depending  upon  the  particular  way  that  the  evaluation 
is  introduced.  In  contrast  to  the  goal  attainment  method  of  evaluation, 
there  is  no  standardized,  single  score  that  represents  the  total 
success  of  the  therapy.  The  lack  of  such  an  index  makes  it  difficult 
to  make  comparisons  between  therapists. 

The  focus  of  problem -oriented  records  is  technique -oriented 
rather  than  accomplishment-oriented.  Keeping  good  records  is  not 
an  end  in  itself  but  the  ultimate  goal  is  to  achieve  a better  quality 
of  service.  Thus,  there  are  some  important  linkages  involved  in 
order  for  better  treatment  to  result  from  better  record-keeping 
including  the  following:  (1)  the  therapists  have  to  be  taught  how  to 
keep  the  records  in  the  new  form;  (2)  they  have  to  carry  out 
the  system  faithfully  rather  than  merely  giving  it  "lip  service";  (3) 
the  records  would  have  to  be  reviewed  regularly  and  recommendations 
made  to  reinforce  good  practices  and  eliminate  bad  practices; 

(4)  the  therapists  would  have  to  carry  out  the  recommendations  made 
on  the  basis  of  the  record  system.  Finally,  the  whole  system 
depends  upon  the  assumption  that  the  record  system  measures  the 
quality  of  service  accurately  enough  for  useful  suggestions  and  better 
practices  to  result  from  it. 
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The  assumption  that  keeping  better  records  leads  to  better 
practices  is  challenged  by  the  director  of  a Center  in  the  following 
exchange : 


(Director)  I feel  that  we  need  to  have  somebody  look 
into  the  efficiency  of  the  organization,  perhaps  staying  a 
month  or  two  in  every  location.  It  might  be  threatening  but 
it  is  likely  to  help. 

(Evaluator)  That  is  already  happening  in  certain  of 
the  areas  but  what  we  need  are  standards  of  practice. 

(Supervisory  Clinician)  I haven't  been  getting  anywhere 
with  the  problem-oriented  records.  In  order  to  put  these 
into  operation,  I feel  that  they  would  have  to  be  standardized 
across  the  different  service  areas. 

(Director)  I feel  that  [names  of  certain  supervisory 
clinicians]  are  doing  good  jobs,  and  that  the  way  that  we  can 
improve  our  quality  is  by  recruiting  people  who  already  have 
experience. 

(Evaluator)  There  should  be  clear  guidelines  and 
regulations  concerning  what  is  expected,  particularly  with 
respect  to  treatment  records. 

(Director)  It  is  hard  to  put  down  guidelines  in  terms  of 
good  treatment.  For  example,  I can  keep  good  records  but 
make  it  perfectly  clear  to  the  patient  that  I don’t  care  what 
he  thinks,  and  still  get  him  out  of  the  office  by  saying  the 
right  things. 

(Evaluator)  Or  you  could  also  have  poor  records  and 
poor  treatment  or  poor  records  and  good  treatment.  . . . 

Whatever  the  reason,  the  problem -oriented  record  system 

apparently  failed  to  take  hold  in  the  two  centers  where  it  was 

introduced.  Part  of  the  failure  was  the  result  of  the  fact  that  only 

a few  persons  in  each  center  were  truly  interested  in  establishing 
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the  system.  Despite  their  formal  powers,45  these  persons  ran 
into  strong  opposition  from  lower  level  persons  who  felt  that  the 
system  was  a burden  on  them: 

(Director)  The  way  that  it  was  implemented  here  was 
that  the  clinical  director  tried  to  influence  the  other  physicians 
to  undertake  the  system.  But  they  weren't  keen  on  it  or 
didn't  understand  it.  When  it  comes  to  dictating  the  patient's 
history  and  the  paperwork,  the  nurses  are  probably  the  ones 
doing  it.  And  so  the  clinical  director  would  dictate  the 
problems  and  discuss  the  solutions  that  he  wanted  done.  But 
the  nurses  were  resistent  to  this.  They  had  been  used  "to 
looking  at  the  record  and  seeing  that  he  wanted  the  patient 
to  have  this  degree  of  privileges,  for  example,  either  to  go 
to  recreation  or  not  to  go.  . . . And  then  one  day  the  nurse 
goes  down  there  and  the  clinician  would  tell  her  to  discharge 
the  patient  which  she  would.  But  here,  with  the  problem  - 
oriented  system,  all  of  a sudden,  you  have  a psychiatrist 
saying,  "Here  is  the  ball,  you  run  with  it.  I made  the  game- 
plan.  ..."  The  nurses  react  such  as  follows:  "How  do  you, 
the  psychiatrist,  know  from  talking  for  thirty  minutes  with  him 
what  he  wants  to  do?  It  is  easy  to  say  to  have  him  quit 
smoking,  but  here  he  is  saying  that  if  I don't  give  him  a cigarette 
that  he  will  kill  me.  " 

Thus,  the  lower  level  personnel  in  the  organization  feel  that 
this  record  system  places  a burden  on  them.  The  difficulty  of 
implementing  this  evaluative  tool  is  revealed  in  the  following  exchanges 
among  a director,  psychiatrist,  and  evaluator  concerning  how  to 


deal  with  the  new  record-keeping  system. 


(Psychiatrist)  How  would  this  be  for  a problem -oriented 
model:  (1)  Problem:  frequent  running  away  from  school, 

(2)  Goal:  to  help  him  remain  in  school,  and  (3)  Plan:  family 
therapy? 

(Evaluator)  That  isn't  specific  enough.  The  goal  should 
be  more  specific  so  that  we  can  evaluate  whether  or  not  it 
was  achieved,  for  example,  decrease  by  50  percent  the 
amount  of  running  away. 


4bA  director  and  clinical  head  were  the  persons  who  attempted 
to  implement  the  system  in  the  two  organizations. 
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(Psychiatrist)  When  I can  get  the  paraprofessionals  to 
say  that  the  problem  is  running  away  from  school  rather  than 
adolescent  rebellion,  I have  achieved  something. 

(Director)  Isn't  there  some  thing  or  some  way  for  the 
data  to  be  computerized  so  that  we  can  get  sequential  records 
for  each  patient  when  he  is  seen? 

(Psychiatrist)  That  is  the  point  we  want  to  get  to,  but 
first  you  have  to  teach  the  people  how  to  collect  the  data. 

(Director)  Then  why  can't  you  have  the  computer  yell 
out  when  this  particular  item  isn't  filled  in? 

(Psychiatrist)  That  would  simply  make  them  go  for 
shallow  answers  and  make  it  mechanical.  You  would  have  to 
audit  the  records.  They  would  need  a professor  at  their  elbow 
to  fill  these  out. 

Here  we  see  the  importance  of  two  of  the  linkages  already  mentioned-- 
the  problem  of  teaching  the  method  and  the  problem  of  having  the 
practitioners  take  it  seriously  enough  to  use  it.  In  the  centers  where 
it  was  introduced,  the  problem -oriented  system  was  gradually 
abandoned.  The  reasons  included  indifference  on  the  part  of  staff 
and  the  more  pressing  demands  of  rising  client  loads  which  would 
only  have  been  aggravated  by  a more  complicated  form  of  record- 
keeping. 


Other  Evaluations 

In  addition  to  the  kinds  of  evaluations  that  we  have  already 
examined,  there  were  other  forms  of  evaluation  being  carried  out. 
However,  they  were  either  limited  to  one;  center  or  failed  to  meet 
the  criteria  established  for  scientific  evaluation.  One  of  the  more 
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interesting  of  these  other  forms  of  evaluation  is  called  "process 

analysis.  " It  was  initiated  at  one  of  the  centers.  Most  evaluations 

attempt  to  measure  the  results  of  therapy  but  process  analysis 

focuses  on  the  interaction  between  therapist  and  client  while  the 

therapy  is  being  carried  out.  At  repeated  intervals,  the  client 

and  therapist  fill  out  questionnaires  concerning  their  subjective 

feelings  and  perceptions  concerning  the  other  person.  Some  evidence 

exists  that  the  degree  of  empathy  that  exists  between  the  client  and 

therapist  is  a major  factor  in  determining  whether  the  outcome  is 
46 

successful.  In  this  particular  case,  the  evaluator  found  some 

interesting  patterns,  for  example,  that  some  therapists  do  better 

at  certain  times  of  the  day.  However,  other  than  providing  some 

interesting  feedback  to  the  clinicians,  this  evaluation  failed  to 

47 

produce  any  substantive  change  in  the  Center's  operations. 

There  were  some  evaluations  which  focused  on  specific 
programs  of  particular  organizations.  An  example  of  this  kind  of 
evaluation  is  the  following  which  concerned  a program  for  emotionally 
disturbed  children: 

4®See  Jerome  D.  Frank,  "The  Bewildering  World  of  Psycho- 
therapy, " JojLirrial_of_Soci^^  , Vol.  28,  No.  4 (1972),  pp.  28-43. 

47 

However,  several  persons,  including  the  evaluator, left 
the  organization  before  this  evaluation  had  been  established  for 
very  long. 


67 


(Evaluator)  We  have  a series  of  tests  given  to  them. 

One  is  a social  adaptation  test  that  is  given  to  the  classroom 
teacher  from  the  regular  classroom  from  which  they  come. 

Then  it  is  given  to  the  classroom  teacher  here  and  they  are  also 
given  a test  for  academic  test  and.  . . for  social  adaptability. 

If  this  works  out,  we  can  use  it  as  an  actual  instrument  to 
decide,  or  aid  in  making  the  decision,  about  when  these 
children  should  return  to  the  school.  . . 

While  this  evaluation  has  potential  implications  for  the  quality  of 

the  program,  there  were  no  plans  to  draw  such  conclusions  but 

rather  to  focus  on  decisions  that  concerned  the  students. 

Finally,  there  were  a number  of  miscellaneous  evaluations 

which  bordered  on  or  clearly  did  not  meet,  the  criteria  necessary 

to  be  called  scientific  evaluations.  These  include  the  following: 

(1)  meetings  at  which  the  treatment  of  a few,  selected  cases  was 

discussed  by  clinicians  in  a program;  (2)  performance  ratings  by 

supervisory  personnel  of  the  organization  concerning  the  therapists 

under  their  control. 

The  use  of  a review  committee  or  general  meetings  to  review 
the  practices  and  effectiveness  of  therapy  is  a common  practice 
in  these  organizations.  When  such  meetings  effectively  combine 
the  judgments  of  qualified  observers,  such  judgments  are  likely  to 
be  more  reliable  than  the  judgments  of  a single  observer.  48  Such 


4^For  a discussion  of  the  problems  of  rating  systems,  whether 
carried  out  by  a group  or  an  individual  supervisor,  see  Lee  J. 
Cronbach,  Essentials  of  Psychological  Testing  (3rd  ed. ; New  York: 
Harper  and  Row,  1970),  pp.  571-607. 
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meetings  often  combine  more  subjective  forms  of  judgment  with 
certain  selected  statistical  data.  A director  describes  the  nature 
of  one  such  meeting  in  a crisis  intervention  program: 


(Director)  Each  week  a meeting  is  held  to  discuss  cases, 
review  procedures,  and  provide  inservice  training  in  order 
to  improve  the  total  program.  Each  week  three  active  cases 
are  picked  for  critical  review.  In  addition,  four  to  twenty-four 
crisis  calls  are  critically  evaluated  and  reviewed.  During 
the  year,  at  various  times,  statistical  data  is  collected  to 
describe  the  number  of  cases,  referral  sources,  reasons 
for  referrals,  and  so  forth. 

It  is  likely  that  in  such  meetings  the  value  of  the  criteria  used 
depends  to  a great  degree  on  the  particular  individuals  heading  the 
programs  and  leading  the  discussions.  Checking  for  the  validity  and 
reliability  of  the  judgments  expressed  is  usually  beyond  the  resources 
of  those  involved  in  the  discussions,  even  if  they  felt  the  need  to 
do  so. 


Rating  by  a superior  is  one  of  the  most  prevalent  forms  of 
evaluation  in  any  type  of  organization.  In  the  organizations  under 
study,  there  is  usually  a checklist  of  items  on  which  the  therapist 
is  rated.  A clinician  at  one  of  the  centers  describes  one  such  list: 

(Supervisory  Clinician)  Quality-wise,  of  course,  each 
clinician  is  evaluated  by  five  areas  that  we  look  at:  (1)  forms, 
scheduling,  and  this  kind  of  thing,  (2)  ability  to  work  with 
other  clinicians,  (3)  the  diagnostic  astuteness  of  the  particular 
person,  and,  if  you  are  going  to  separate  diagnosis  from 
treatment,  (4)  their  actual  treatment  ability. 


The  validity  and  reliability  of  such  rating  systems  is  problematic, 
being  affected  by  personal  relationships,  "halo  effects,  " 


rumors. 
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and  many  other  nonscientific  factors.  49 

Summary 

Each  major  evaluative  technique  used  by  these  organizations 
has  been  analyzed  with  respect  to  the  scope  of  the  activities  covered 
by  the  evaluation,  the  degree  to  which  it  is  accomplishment-oriented, 
and  the  extent  to  which  it  has  had  a substantive  impact  on  the  operations 
of  these  organizations. 

Goal  attainment  scaling  and  problem -oriented  records  both 
have  a potentially  broad  scope  because  they  can  be  applied  to  nearly 
any  service  offered  by  the  organization.  Consumer  satisfaction 
surveys  and  needs  assessment  surveys  appear  generally  to  have  too 
broad  a scope  to  be  able  tc  provide  much  information  that  can  lead 
to  specific  recommendations.  None  of  these  evaluations  covered, 
or  was  even  related  to,  important  decision  areas  such  as  budgetary 
and  personnel  decisions. 

The  consumer  satisfaction  surveys  focused  upon  an  importanc 
goal  of  the  satisfaction  of  the  clientele.  However,  it  was  found 
that  these  surveys  did  not  provide  information  that  discriminates 
clearly  between  good  and  poor  services.  The  goal  attainment  forms 
of  evaluation  have  a sharper  focus  because  they  measure  the  extent 
to  which  clients  are  perceived  to  have  attained  specific  goals. 

49A1so,  a tendency  to  give  only  high  ratings  usually  emerges  in 
such  a system.  See  Peter  M.  Blau,  The  Dynamics  of  Bureaucracy, 
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Psychiatric  epidemiological  surveys  also  focus  upon  an  important 
goal  of  these  organizations --the  decrease  in  the  prevalence  of  mental 
illness  in  the  areas  served  by  the  organizations.  However,  these 
surveys  provide  information  that  is  far  from  clear  concerning  the 
quality  and  effectiveness  of  the  services  offered  by  the  organization. 
Also,  it  generally  takes  an  extremely  long  period  of  time  before 
any  feedback  is  derived  from  these  surveys.  Problem -oriented 
records  are  directed  at  a more  technique -oriented  goal  of  these 
organizations,  the  keeping  of  good  records.  As  a consequence,  the 
expected  impact  of  keeping  good  records  depends  upon  several 
assumptions  concerning  the  relationship  between  such  records  and 
the  ultimate  effectiveness  of  the  treatment. 

Distinguished  by  their  absence  are  evaluations  which  focus 
on  the  effect  these  organizations  have  on  decreasing  the  prevalence 
of  behaviors  such  as  alcoholism  and  drug  addiction.  No  attempt  was 
made  to  use  controls  in  the  evaluation,  either  through  the  use  of 
experimental  or  quasi -experimental  designs.  Also,  none  of  these 
evaluations  focuses  upon  the  success  of  programs  as  opposed  to 
individual  therapists.  The  scores  obtained  from  the  goal  attainment 
method  can  be  aggregated  to  form  a score  for  an  entire  program 
but  the  focus  of  the  evaluation  remains  the  work  of  the  individual 
therapist. 

In  regard  to  the  impact  of  these  evaluations,  certain  types  of 
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evaluations  such  as  goal  attainment  evaluation  provided  feedback 
to  therapists.  Whether  this  added  feedback  actually  led  to  increased 
quality  in  the  services  offered  by  the  organizations  was  not  tested. 
Psychiatric  epidemiological  surveys  were  used  to  help  determine 
the  location  of  some  new  services. 

Aside  from  the  above  examples,  these  forms  of  evaluation 
appear  to  have  had  strikingly  little  substantive  effect  on  the  operations 
of  these  organizations.  50  Indeed,  many  of  the  directors  of  these 
organizations  were  undertaking  evaluations  such  as  consumer 
satisfaction  surveys  and  epidemiological  surveys  from  which  they 
expected  to  obtain  little,  if  any,  useful  information.  51  They 
expected  these  evaluations  to  confirm  only  what  they  already  knew. 

All  forms  of  evaluations  studied  consumed  significant  amounts 
of  resources  such  as  the  time  of  the  clinical  staff.  Thus,  because 
of  the  time  spent  in  carrying  out  psychiatric  epidemiological  surveys 
or  filling  out  goal  attainment  forms,  the  clinicians  may  be  unable  to 
see  as  many  clients. 


50These  conclusions  again  must  be  qualified  for  the  following 
reasons.  (1)  most  evaluations  had  not  been  in  existence  for  a 
great  period  of  time;  (2)  the  evidence  is  based  on  interviews  and 
participant  observation. 

51 

The  author  is  referring  to  internal  organizational  uses 
only.  In  later  chapters,  other,  externally-oriented  functions  of 
these  evaluations  are  examined. 
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The  general  lack  of  much  impact  would  tend  to  support 
Wildavsky’s  contention  that  organizations  cannot  evaluate  them- 
selves critically.  However,  this  conclusion  is  tentative  and  leaves 
unanswered  several  important  questions  that  need  to  be  explored 
in  future  chapters,  including  the  following:  (1)  Is  the  lack  of  impact 
of  these  evaluations  the  result  of  resistance  of  the  therapists  to 
these  methods  of  formal  evaluation?  (2)  How  are  decisions  made 
if  they  are  not  decided  on  the  basis  of  formal,  scientific  evaluations? 
These  and  other  issues  will  be  explored  in  the  next  chapter,  in 
which  attitudes  towards  evaluation  are  examined. 


CHAPTER  III 


ATTITUDES  TOWARD  EVALUATION 


The  purpose  of  this  chapter  is  to  examine  attitudes  toward 
evaluation  held  by  personnel  of  the  organizations  under  study.  In 
particular,  the  attitudes  of  three  groups  of  personnel,  whose  acceptance 
of  formal  evaluation  is  critical  to  its  success,  are  examined: 
clinicians,  directors,  and  evaluators. 

In  Chapter  II,  it  was  concluded  that  formal  evaluation  has 
brought  about  little  change  in  the  operations  of  the  mental  health 
organizations  in  which  it  has  been  introduced.  1 In  Chapter  I, 
three  problem  areas  of  organizational  theory  which  provide  ex- 
planations as  to  why  evaluation  is  likely  to  have  little  impact  were 
introduced: 

(1)  The  relationship  between  the  natural  and  artificial 
systems  of  an  organization:  the  development  of  a formalized 
evaluation  system  may  be  viewed  as  an  infringement  on  the 
natural  system  elements  of  the  organization. 

(2)  The  technology  of  evaluation:  resistance  to  evalua- 
tion may  be  the  result  of  the  lack  of  development  of  the 
technology  of  evaluation  and  result  in  skepticism  among  the 
personnel  of  the  organizations  toward  the  evaluation. 


■*"As  noted  in  Chapter  II,  this  conclusion  is  qualified  on  the 
basis  of  the  particular  time  period  when  these  fourteen  organizations 
were  studied. 
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(3)  The  struggle  for  survival:  the  necessity  to  put 
survival  needs  first  may  block,  the  evaluation  from  having 
any  significant  impact  on  the  organization. 

Through  the  examination  of  the  attitudes  toward  evaluation, 

the  importance  of  the  first  two  of  the  above  factors  is  examined 

in  regard  to  their  being  obstacles  to  the  successful  implementation 

of  formal  evaluation.  In  other  words,  can  the  apparent  lack  of  impact 

of  evaluation  be  attributed  to  the  clinicians'  anxiety  and  resistance 

to  the  introduction  of  formal  evaluation?  Is  the  lack  of  impact  of 

evaluation  the  result  of  skepticism  concerning  the  validity  and 

reliability  of  the  technology  of  formal  evaluation? 

To  aid  in  our  analysis  of  attitudes  toward  formal  evaluation, 

attitudes  toward  informal  and  openly  subjective  types  of  evaluation 

are  also  studied.  Likewise,  comparisons  are  made  between  the  use 

of  information  derived  from  formal  and  informal  types  of  evaluation 

in  the  decision  making  of  the  organizations  under  study. 

Attitudes  Toward  Formal  Evaluation 
The  exploration  of  the  attitudes  toward  evaluation  is  based 
a combination  of  participant  observation,  interviews,  and  questionnaires. 
The  questionnaire  consists  of  twenty-eight  questions  designed  to 
explore  significant  issues  relevant  to  analyzing  the  attitudes  toward 
formal  evaluation  held  by  clinicians,  directors,  and  evaluators 
within  these  fourteen  organizations.  The  items  included  in  the 


questionnaire  were  selected  as  relevant  based  on  a search  of  the 
literature,  five  months  of  participant  observation,  and  interviews 
with  evaluators  in  different  organizations.  The  total  number 
of  persons  included  in  the  questionnaire  survey  is  small.  Non- 
random  methods  of  selection  were  used  due  to  the  limitations  of 
time,  access,  and  money  in  the  selection  of  the  respondents.  Thus, 
inferences  beyond  the  particular  group  of  persons  sampled  are 
problematical.  The  questionnaire  is  intended  to  serve  only  as 
an  approximation  of  the  attitudes  of  the  individuals  sampled  and 
a supplement  to  the  information  derived  from  participant  observation 


2 

Thus,  the  questionnaire  was  not  constructed  until  after 
five  months  of  participant  observation  had  already  been  carried  out. 
A preliminary  form  of  the  questionnaire  was  given  to  the  evaluators 
who  offered  suggestions  that  were  incorporated  into  the  final  form. 
It  should  be  noted  that  the  instructions  to  the  questionnaire  specified 
that  the  questions  referred  specifically  to  formal  evaluation  carried 
out  by  an  evaluator  and  his  staff. 

3 

Questionnaires  were  given  to  all  directors  and  evaluators 
within  the  fourteen  organizations  at  the  time  of  the  study.  In  two 
organizations,  there  were  two  persons  identifiable  as  directors. 

In  one  case,  directorial  responsibilities  were  divided  between  a 
clinical  and  administrative  director.  In  another  case,  there  was 
a change  of  directors  during  the  course  of  the  study.  Question- 
naires were  given  to  clinicians  on  the  basis  of  access  in  each  of 
the  organizations.  Thus,  there  may  be  a bias  in  the  responses 
obtained.  However,  the  questionnaires  are  intended  to  serve  (as 
is  noted  above)  only  as  a supplementary  form  of  information  and 
the  data  derived  from  them  were  generally  supported  by  inter- 
views and  participant  observation  carried  out. 


76 


and  interviews. 
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Natural  System  Elements 

The  natural  system  of  an  organization  consists  of  unprogrammed 
elements  such  as  personal  friendships,  cliques,  and  informal  group 
values.  5 Decisions  concerning  such  personal  matters  as  job 
promotion  and  security  are  determined  largely  by  informal  and 
subjective  judgments  which  are  closely  involved  with  the  elements 
that  compose  the  natural  system.  The  use  of  formal  evaluation 
carried  out  by  an  evaluator  is  likely  to  be  viewed  as  an  infringement 
upon  the  elements  of  the  natural  system  of  the  organization  if  the 
informal  system  is  a legitimate  obstacle  to  the  successful  im- 
plementation of  formal  evaluation. 

To  explore  the  importance  of  these  natural  system  elements 
through  the  use  of  questionnaires,  it  was  hypothesized  that  clinicians 
would  favor  evaluations  that  were  highly  generalized  and  non- 
threatening but  would  oppose  evaluations  that  concerned  matters  of 
more  direct  importance  to  themselves  such  as  job  promotion  and 


See  the  appendix  for  a complete  listing  of  the  questions 
asked.  The  first  and  second  halves  of  the  questionnaire  are  intended 
to  be  equivalent  measures  of  attitudes  toward  evaluation.  An  index 
of  reliability  of  the  questionnaire,  computed  according  to  Rulon's 
formula,  was  calculated  to  be  . 97  based  on  the  sixty-three  completed 
questionnaires  (of  the  original  eighty-four  sent  out).  See  J.  P. 
Guilford,  Psychometric  Methods  (New  York:  McGraw -Hill  Book 
Company,  1954),  pp.  385-386. 

5 

Thus,  the  term  "natural  system"  is  an  analytical  concept. 
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security- -matters  that  are  now  mainly  determined  by  informal 
judgments. 

Responses  to  the  items  in  Tables  3.  1 through  3.  9 tend  to 
confirm  the  thesis  that  clinicians  support  highly  general  evaluations 
but  are  much  less  enthusiastic  about  evaluations  that  concern  matters 
of  personal  importance  to  them.  Tables  3.  1 and  3.  2 reveal  that 
approximately  90  percent  or  more  of  the  clinicians  support  evaluation 
for  the  general  purposes  of  providing  feedback  to  the  clinical  staff 
and  for  carrying  out  an  overall  evaluation  of  the  organization. 

Tables  3.  3,  3.  4,  and  3.  5 reveal  strong  support  also  for  the  concept 
of  program  evaluation  with  well  over  80  percent  of  the  clinicians 
favoring  evaluation  for  this  purpose  in  each  case. 

However,  support  drops  sharply  for  formal  evaluation  in 
the  items  relating  to  evaluation  of  individual  performances  in 
Tables  3.  6,  3.  7,  3.  8,  and  3.  9.  More  specifically,  the  items  in 
Tables  3.  6 and  3.  7 concerning  the  use  of  evaluation  to  judge  whether 
a membenof  the  organization  is  doing  a good  job  receive  approxi- 
mately 60  and  50  percent  support  respectively  from  clinicians.  The 
two  items  concerning  the  use  of  formal  evaluation  to  determine  who 
should  be  promoted  or  fired  receive  only  34  and  21  percent  support 
respectively.  ^ 

0 

The  repetitiveness  of  many  of  the  questions  results  from  the 
use  of  the  split-half  method  of  measuring  reliability.  In  this  method, 
two  approximately  equal  halves  of  a questionnaire  are  constructed. 
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TABLE  3.  1 
FEEDBACK 


Evaluation  can  and  should  be  used  to  furnish  feedback  for  the 
clinicial  staff. 

Response 

Percentage  of 
Clinicians 

Percentage  of 
Directors 

Percentage  of 
Evaluators 

Strongly  Agree 

73.  7 (28) 

75. 0 (12) 

100  (9) 

Agree 

23. 7 (9) 

25.  0 (4) 

— 

Undecided 

2.  6 (1) 

— 

— 

Disagree 

— 

— 

— 

Strongly  Disagree 

— 

— 

— 

TABLE  3.  2 

OVERALL  ORGANIZATION 

Evaluation  can  and  should  be  used  to  determine  whether  or 
not  the  overall  organization  is  performing  well. 


Response 

Percentage  of 
Clinicians  ' 

Percentage  of 
Directors'1 

Percentage  of 
Evaluators 

Strongly  Agree 

36.  8 (14) 

43. 8 (7) 

66.  7 (6) 

Agree 

52.  6 (20) 

43.  8 (7) 

33.  3 (3) 

Undecided 

10.  5 (4) 

6.  3 (1) 

— 

Disagree 

— 

6.  3 (1) 

— 

Strongly  Disagree 

— 

— 

— 

'Burn  of  percentage  figures  is  unequal  to  100  percent  because  of 


rounding. 
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TABLE  3.  3 

PROGRAM  EVALUATION:  #1 


Evaluation  can  and  should  be  used  to  determine 
the  organization's  programs  on  the  community. 

the  impact  of 

Response 

Percentage  of 
Clinicians 

Percentage  of 
Directors* 

Percentage  of 
Evaluators 

Strongly  Agree 

28.  9 (11) 

18.  8 (3) 

55.  6 (5) 

Agree 

63.  2 (24) 

56.  3 (9) 

44.  4 (4) 

Undecided 

5.  3 (2) 

18.  8 (3) 

— 

Disagree 

2.  6 (1) 

6.  3 (1) 

— 

Strongly  Disagree 

— 

— 

— 

*Sum  of  percentage  figures  is  greater  than  100  percent  because  of 
rounding. 


TABLE  3.  4 

PROGRAM  EVALUATION:  #2 


Evaluation  can  and  should  be  used  to  determine  which  service 
programs,  if  any,  are  performing  poorly. 


Response 

Percentage  of 
Clinicians  * 

Percentage  of 
Directors* 

Percentage  of 
Evaluators 

Strongly  Agree 

28.  9 (11) 

25.  0 (4) 

22. 2 (2) 

Agree 

57. 9 (22) 

50. 0 (8) 

77. 8 (7) 

Undecided 

10.  5 (4) 

6.  3 (1) 

— 

Disagree 

2.  6 (1) 

18.  8 (3) 

— 

Strongly  Disagree 

— 

— 

— 

*Sum  of  percentage  figures  is  unequal  to  100  percent  because  of 
rounding. 
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TABLE  3.  5 

PROGRAM  EVALUATION:  #3 


Evaluation  can  and  should  be  used  to  determine  whether 
individual  mental  health  programs  are  successful  or  not. 

Response 

Percentage  of 
Clinicians  * 

Percentage  of 
Directors  * 

Percentage  of 
Evaluators 

Strongly  Agree 

39.  5 (15) 

43. 8 (7) 

88.  9 (8) 

Agree 

47.  4 (18) 

50. 0 (8) 

11. 1 (1) 

Undecided 

13.2  (5) 

— 

— 

Disagree 

— 

6.  3 (1) 

— 

Strongly  Disagree 

— 

— 

— 

*Siim  of  percentage  figures  is  greater  than  100  percent  because  of 
rounding. 


TABLE  3.  6 

EVALUATION  OF  INDIVIDUAL  CLINICIANS:  #1 


Evaluation  can  and  should  be  used  to  determine  whether  or  not 
individual  clinical  staff  members  are  performing  their  jobs  well. 


Response 

Percentage  of 
Clinicians 

Percentage  of 
Directors* 

Percentage  of 
Evaluators* 

Strongly  Agree 

15.  8 (6) 

18.  8 (3) 

11.1  (1) 

Agree 

47.  4 (18) 

50. 0 (8) 

22. 2 (2) 

Undecided 

10.  5 (4) 

12.  5 (2) 

44.  4 (4) 

Disagree 

26.  3 (10) 

18.  8 (3) 

22. 2 (2) 

Strongly  Disagree 

— 

— 

— 

*Sum  of  percentage  figures  is  unequal  to  100  percent  because  of 
rounding. 
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TABLE  3.  7 

EVALUATION  OF  INDIVIDUAL  CLINICIANS:  #2 


Evaluation  is  neither  valid  nor 
determine  who  is  doing  a good  job. 

reliable  enough  to  be  used  to 

Response 

Percentage  of 
Clinicians 

Percentage  of 
Directors- 

Percentage  of 
Evaluators 

Strongly  Agree 

— 

6.  3 (1) 

— 

Agree 

26.  3 (10) 

31. 3 (5) 

33.  3 (3) 

Undecided 

23. 7 (9) 

31.  3 (5) 

— 

Disagree 

47.  4 (18) 

18.  8 (3) 

66.  7 (6) 

Strongly  Disagree 

2.  6 (1) 

12.  5 (2) 

— 

-Sum  of  percentage  figures  is  greater  than  100  percent  because  of 
rounding. 


TABLE  3.  8 

EVALUATION  OF  INDIVIDUAL  CLINICIANS:  #3 

Evaluation  is  currently  neither  valid  nor  reliable  enough  to  be 
used  to  determine  who  should  be  promoted  within  the  organization. 


Response 

Percentage  of 
Clinicians 

Percentage  of 
Directors- 

Percentage  of 
Evaluators- 

Strongly  Agree 

15.  8 (6) 

25.  0 (4) 

22. 2 (2) 

Agree 

36.  8 (14) 

12. 5 (2) 

22. 2 (2) 

Undecided 

13.  2 (5) 

43. 8 (7) 

22. 2 (2) 

Disagree 

34.2  (13) 

12. 5 (2) 

33. 3 (3) 

Strongly  Disagree 

— 

6.  3 (1) 

— 

-Sum  of  percentage  figures  is  unequal  to  100  percent  because  of 
rounding. 
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TABLE  3.  9 

EVALUATION  OF  INDIVIDUAL  CLINICIANS:  #4 


Evaluation  can  and  should  be  used  to  determine 
fired  from  their  jobs. 

who  should  be 

Response 

Percentage  of 
Clinicians 

Percentage  of 
Directors  * 

Percentage  of 
Evaluators* 

Strongly  Agree 

— 

— 

--- 

Agree 

21. 1 (8) 

12. 5 (2) 

— 

Undecided 

28.  9 (11) 

31. 3 (5) 

33. 3 (3) 

Disagree 

34.  2 (13) 

43. 8 (7) 

22. 2 (2) 

Strongly  Disagree 

15.  8 (6) 

12.  5 (2) 

44.  4 (4) 

*Sum  of  percentage  figures  is  unequal  to  100  percent  because  of 
bounding. 


In  short,  the  thesis  that  clinicians  favor  generalized,  formal 
evaluations  more  than  formal  evaluations  which  focus  on  individuals 
is  supported  by  the  above  evidence.  The  data  offer  support  for  the 
view  that  the  elements  composing  the  natural  system  present  an 
obstacle  to  formal  evaluation.  In  other  words,  the  relatively  low 
support  found  for  the  use  of  formal  evaluation  as  a means  of  evaluating 
individuals  can  be  interpreted  as  indicating  that  clinicians  wish  to 
keep  such  evaluation  more  informal  and  personalized. 

Two  qualifications  must  be  made  concerning  this  conclusion. 
First,  despite  the  fact  that  support  for  individualized  uses  of  formal 
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evaluation  is  lower  than  more  general  uses  of  evaluations,  fairly 
large  percentages  of  the  clinicians  in  the  sample,  34  percent  and  21 
percent  respectively,  support  the  use  of  formal  evaluation  for  even 
the  most  threatening  of  uses  such  as  promotion  and  firing.  Based 
on  participant  observation  and  interviews,  the  reason  for  the  support 
for  formal  evaluation  to  be  used  for  such  functions  can  be  explained 
by  the  fact  that  not  all  clinicians  are  happy  with  the  way  in  which 
individual  performances  are  now  judged.  ^ One  evaluator  described 
a situation  in  which  a clinician  felt  "put  down"  by  the  judgments 
of  his  superiors  and  other  staff  in  the  organization.  This  situation 
led  the  clinician  to  ask  to  be  evaluated  by  the  use  of  formal,  goal- 
attainment  scaling  evaluation: 

(Evaluator)  I don't  know  why  but  several  times  [name 
of  the  clinician]  was  put  down.  People  were  saying  that  he 
simply  wasn't  doing  anything  worthwhile.  So,  he  said. 

Alright,  I want  to  be  evaluated.  Come  over  and  evaluate 
us.  It  was  like,  "We're  open  to  anything.  " 

Thus,  some  clinicians  support  formal  evaluation  for  even  the  most 
threatening  of  uses. 

A second  qualification  is  that  the  responses  of  the  clinicians 
need  not  be  interpreted  simply  as  a preference  for  purely  subjective 


n 

The  preference  of  French  workers  for  avoiding  situations 
in  which  their  supervisors  have  discretionary  power  over  them  has 
been  pointed  out  by  Michel  Crozier  in  The  Bureaucratic  Phenomenon 
(Chicago:  University  of  Chicago  Press,  1964). 
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evaluation.  Clinicians  most  strongly  support  evaluations  which 
give  them  something  in  return  such  as  feedback.  Likewise,  they 
offered  greatest  opposition  to  the  use  of  evaluation  for  the  most 
threatening  decision  of  all- -the  decision  to  fire  someone  from  a job. 
Thus,  the  responses  of  the  clinicians  may  be  interpreted  within  the 
framework  of  a simple  decision  making  model:  support  evaluations 
that  do  not  threaten  your  own  position  and  that  offer  you  potentially 
positive  rewards  such  as  feedback. 

It  might  be  expected  that  directors  would  be  more  favorable 
toward  the  use  of  formal  evaluation  to  evaluate  individual  clinicians. 
They  are  at  the  top  of  the  formal  authority  structure  and  would  have 
less  reason  to  be  anxious  about  the  use  of  formal  evaluation  for  this 
purpose.  However,  Tables  3.  1 through  3.  9 generally  show  that 
the  responses  of  the  directors  parallel  those  of  the  clinicians.  The 
directors  are  much  more  favorable  towards  the  use  of  formal 

evaluation  for  general  purposes  than  for  the  evaluation  of  individual 
clinicians. 

Drawing  on  interviews  and  participant  observation,  the  failure 
of  the  directors  to  be  any  more  favorable  than  the  clinicians  toward 
the  use  of  evaluation  for  this  purpose  can  be  due  to  two  reasons. 
First,  many  of  the  directors  retain  a strong  identification  with  the 
clinical  role  and  the  point  of  view  of  the  individual  clinician. 

Secondly,  while  they  are  not  likely  to  be  personally  threatened  by 
such  evaluation,  nevertheless  the  formal  evaluation  would  take  away 
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some  of  their  power.  Their  informal  judgments  would  no  longer 
be  the  sole  way  of  evaluating  the  personnel  under  their  control. 

The  responses  of  the  evaluators  provide  more  substantial 
evidence  for  the  validity  of  the  natural  system  as  a constraint  on 
evaluation.  In  Tables  3.1  through  3.5,  the  evaluators  demonstrate 
almost  total  support  for  evaluation  which  deals  with  the  overall 
organization,  programs,  and  the  purpose  of  providing  feedback  to 
the  clinicians.  However,  support  for  the  use  of  formal  evaluation 
to  evaluate  individual  performances  drops  off  sharply  among  the 
evaluators  in  Tables  3.  6 through  3.  9.  In  particular,  most  evaluators 
do  not  favor  the  use  of  formal  evaluation  for  the  determination  of  the 
promotion  or  firing  of  individuals.  The  responses  of  the  evaluators 
do  not  make  much  sense  if  interpreted  within  the  framework  of  a 
short-run  power  maximization  model  of  man.  If  the  evaluator  could 
use  formal  evaluation  to  evaluate  the  individual  performances  of 
organizational  members,  he  would  immediately  become  a very 
powerful  figure  in  the  organization. 

Yet,  interviews  and  participant  observation  demonstrated 
that  evaluators  generally  prefer  a much  more  innocuous  form  of 
evaluation.  They  anticipate  great  anxiety,  fear,  and  general 
resistance  to  any  attempt  to  use  formal  evaluation  to  evaluate 
individual  performances.  The  author  asked  one  evaluator  if  he 
distinguished  between  the  evaluation  of  programs  and  the  evaluation 
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of  individuals  and  received  the  following  response: 

(Evaluator)  Yes,  program  evaluation  is  the  basic  unit 
here.  We  will  take  a program  like  the  outpatient  clinic.  . . . 
Measuring  the  success  of  the  individual  therapists  within  that 
clinic  is  possible  using  similar  methodologies.  But  I don't 
think  I would  take  the  analysis  down  that  way  unless  somebody 
asked  me.  . . . 

The  evaluator  was  then  asked  what  advantages  he  saw  in  program 

evaluation  over  the  evaluation  of  individuals: 

(Evaluator)  Well,  it  is  less  threatening.  . . . 

Another  evaluator  expressed  himself  in  a similar  manner  concerning 

his  preference  for  program  evaluation: 

(Evaluator)  Well,  my  personal  preference  is  for  program 
evaluation  but,  unfortunately,  if  a program  flops,  it  is  pretty 
obvious  that  somebody  is  at  fault  there.  I would  rather  not 
have  to  get  down  to  evaluating  specific  people.  I think  that 
is  why  I have  encountered  so  little  resistance  because  I have 
made  the  fact  clear  that  I would  not  be  evaluating  people, 
at  least  for  the  time  being,  if  at  all.  . . . 

In  short,  most  evaluators  do  not  want  to  undertake  evaluations 
that  can  be  viewed  as  threatening  by  persons  they  have  to  work 
with  daily  and  whose  cooperation  they  need  to  implement  even 
non -threatening  evaluations.  ® 

However,  there  is  an  irony  concerning  the  preference  for  more 


O 

Some  evaluators  indicated  that  they  hoped  to  have  more 
of  an  impact  in  the  future.  This  point  is  discussed  further  in  a 
later  chapter. 
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generalized  evaluations  such  as  programs  evaluation.  In  Chapter  II, 
it  was  discussed  that  current  evaluation  techniques  either  focus  on 
the  performance  of  the  individual  therapist  or  are  too  broad  to  draw 
implications  concerning  the  performance  of  programs.  Many 
important  choices  among  types  of  programs  have  to  be  made,  such 
as  whether  to  emphasize  direct  or  indirect  services,  short  or  long 
term  therapy,  and  group  or  individual  therapy.  But  the  evaluative 
techniques  currently  being  used  by  these  organizations  do  not  have 
the  capability  of  evaluating  which  types  of  programs  are  more 
effective. 

Interviews  and  participant  observation  indicated  that  there  was 
ambivalence  concerning  the  proper  focus  of  evaluation.  Most 
clinicians,  evaluators,  and  directors  preferred  the  more  generalized 
evaluations  such  as  program  evaluation.  However,  directors  admit 
in  the  following  responses  that  the  successful  performance  of  the 
individual  clinician  is  a key  to  the  success  of  the  overall  program: 

(Director  #1)  I don't  look  at  the  individual  therapist. 

I look  at  what  you  do  with  your  program.  But  I guess  that 
you  are  evaluating  all  of  the  therapy  that  is  taking  place  in 
the  program,  but  I have  never  thought  of  it  in  terms  of 
evaluating  the  individual  therapists --which  may  be  a natural 
step  but  I haven't  gotten  there. 

(Director  #2)  Sure,  the  real  focus  of  the  evaluation 
should  be  the  interaction  between  the  client  and  therapist. 

That  is  the  program,  generally.  The  key  factor  is  the  quality 
of  the  therapist. 

In  the  following  interview,  a dispute  concerning  the  proper 
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focus  of  evaluation  occurred  between  a clinical  psychologist  and 
a psychiatric  social  worker,  the  former  preferring  program 
evaluation  and  the  latter,  evaluation  of  individuals: 

(Clinical  Psychologist)  Programs  are  a lot  more  solid 
to  deal  with. 

(Social  Worker)  Much  more  what? 

(Clinical  Psychologist)  Concrete  to  deal  with  than 
therapists. 

(Social  Worker)  I would  totally  disagree  with  you.  I 
would  say  that  it  was  the  other  way  around  because  the  only 
thing  that  I know  how  to  evaluate  are  personal  performances 
which  are  easy  for  me  to  evaluate. 

(Clinical  Psychologist)  Subjectively. 

(Social  Worker)  Uh,  in  part. 

(Clinical  Psychologist)  I can  make  a subjective 
evaluation  of  psychologists  in  regard  to  my  confidence  in 
them --all  in  my  head. 

(Social  Worker)  So  far  as  evaluating.  . . I don't  know 
that  I know  of  anything  where  one  can  consciously  set  up  a 
method  of  evaluating  programs. 

(Clinical  Psychologist)  I think  it  might  be  easier  to 
set  up  a set  of  objective  criteria,  for  example,  to  compare  the 
performance  of  social  workers  and  psychologists  with  the 
rating  scale  that  I have  in  mind. 

Thus,  there  exists  ambivalence  concerning  the  proper  focus  of 

formal  evaluations.  Despite  the  fact  that  popular  methods  of  formal 

evaluation,  such  as  goal  attainment  scaling,  are  most  clearly 

directed  at  evaluating  individual  performances,  interviews  and 

questionnaire  data  demonstrate  that  most  clinicians  and  evaluators 
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favor  program  evaluations  over  evaluations  aimed  at  individuals. 

The  evaluators  are  deliberately  trying  to  avoid  threatening 
the  clinical  staff  by  their  evaluations.  Interviews  indicate  that  they 
anticipated  resistance  from  the  elements  of  the  natural  system  of 
the  organizations  if  they  attempted  to  implement  threatening 
evaluations.  At  a meeting  attended  by  both  evaluators  and  state 
officials  bent  on  encouraging  evaluation,  there  was  a disagreement 
between  whether  to  go  slowly  and  avoid  threatening  evaluations 
or  to  begin  by  evaluating  individual  clinicians: 

(State  Official)  I can't  imagine  pressure  being  put  on 
the  therapist.  I think  that  you  will  find  that  most  anxiety 
is  a problem  before  evaluation  is  implemented- -if  it  isn't 
used  to  club  them. 

(Evaluator  #1)  I found  the  same  thing:  resistance  before 
the  implementation  of  evaluation.  . . . 

(Evaluator  #2)  But  I was  placed  on  a probationary 
period  for  my  first  few  months  at  the  . . . Center  and  I 
don't  see  why  evaluation  can't  be  used  to  evaluate  the  staff. 

(State  Official)  It  is  a good  idea.  But  it  is  a problem 
of  not  threatening  or  frightening  people  too  much.  You  have 
to  use  tact.  If  it  is  done  in  a cooperative  way.  . . . 

Due  to  the  fact  that  most  evaluators  prefer  to  introduce  formal 

evaluation  in  a non -threatening  way,  the  influence  of  the  natural 

system  elements  of  the  organization  is  more  indirect  than  direct. 

Although  active  resistance  from  the  informal  elements  did  not  emerge, 

most  evaluators  had  clearly  anticipated  such  resistance,  which  is 

one  of  the  major  reasons  why  they  favored  general  evaluations  such 


as  program  evaluation. 
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The  Technology  of  Evaluation 

How  important  is  skepticism  about  the  technology  of  formal 
evaluation  as  an  obstacle  to  its  implementation?  One  of  the  most 
commonly  stated  arguments  of  those  who  oppose  formal  evaluation 
is  the  contention  that  it  is  impossible  to  quantitatively  measure  the 
crucial  aspects  of  services  to  clients.  An  item  which  formally 
stated  this  argument  was  included  in  the  questionnaire.  Data  in 
Table  3. 10  show  that  slightly  over  30  percent  of  the  clinicians  in 
the  sample  agree  with  this  argument  but  that  over  50  percent  of  the 
clinicians  reject  it. 


TABLE  3.10 

QUANTITATIVE  MEASURES  OF  DIRECT  SERVICES 


It  is  impossible  to  measure  quantitatively  the  most  important 
effects  of  mental  health  direct  services. 


Response 

Percentage  of 
Clinicians 

Percentage  of 
Directors* 

Percentage  of 
Evaluators 

Strongly  Agree 

7.  9 (3) 

— 

— 

Agree 

23. 7 (9) 

37.  5 (6) 

— 

Undecided 

15.  8 (6) 

18.  8 (3) 

11.1  (1) 

Disagree 

44.  7 (17) 

18.  8 (3) 

66. 7 (6) 

Strongly  Disagree 

7.  9 (3) 

25. 0 (4) 

22.  2 (2) 

*Sum  of  percentage  figures  is  greater  than  100  percent  because  of 
rounding. 
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Clinicians,  evaluators,  and  directors  were  all  asked  to  rank 
the  most  important  obstacles  to  the  successful  implementation  of 
a formal  evaluation.  An  index  of  the  relative  importance  of  each 
obstacle  was  constructed  as  described  in  Table  3. 11.  The  data  in 
this  table  reveal  that,  among  clinicians,  the  lack  of  reliability 
and  validity  of  current  evaluative  techniques  was  ranked  as  the  second 
most  important  obstacle  after  the  lack  of  resources  of  time,  money 
and  staff.  It  was  noted  previously  that  clinicians  demonstrated 
strong  support  for  generalized  forms  of  evaluation.  However,  their 
responses  to  the  item  in  Table  3.  11  show  that  they  hold  substantial 
reservations  about  the  technology  of  formal  evaluation.  The  attitudes 
of  many  of  the  clinicians  spoken  with  are  similar  to  those  expressed 
by  the  following. clinician: 

(Clinician)  Well,  I am  not  too  impressed  with  the 
evaluation  methodologies.  But  I think  that  they  are 
something  that  can  be  lived  with.  By  living  with  them, 

I mean  that  you  know  the  pitfalls  of  the  methods. 

In  an  interview  with  two  therapists,  one  argued  that  the  current 

evaluation  technology  was  insufficient.  The  other  therapist  tended 

to  agree  but  argued  that  the  lack  of  evaluation  technology  was  the 

result  of  the  low  priority  given  to  it: 

(Clinical  Psychologist)  I think  that  it  [formal  evaluation] 
could  be  done.  . . that  one  of  the  reasons  that  the  technology 
isn't  better  is  because  we  haven't  been  committed  to  it. 

But  given  the  time  and  money  it  would  take,  who  is  going  to 
pay  for  it  and  where  is  the  payoff  from  it? 
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TABLE  3.  11 

OBSTACLES  TO  EVALUATION 


Please  rank  the  following  factors  in  what  you  regard  to  be 
their  order  of  importance  as  obstacles  to  evaluation.  Evaluation 
here  refers  specifically  to  formal  evaluation  carried  out  by  an 
evaluator  and  his  staff. 

Obstacle  Number: 

1 The  lack  of  reliability  and  validity  of  the  current 
evaluation  techniques. 

2 Anxiety  on  the  part  of  the  direct  service  staff  as  to 
how  evaluation  is  to  be  used. 

3 The  belief  of  professionals  that  they  should  evaluate 
themselves  and  do  not  need  formal  evaluation  by  an 
evaluator. 

4 The  lack  of  skills  necessary  in  the  evaluation  staff 
to  carry  out  a valid  and  reliable  evaluation. 

5 The  lack  of  resources  of  time,  money,  and  staff  for 
the  researchers  to  carry  out  valid  and  reliable  evalua- 
tions. 

The  following  index  of  "importance"  was  formed  by  scoring 
five  points  for  each  time  that  an  obstacle  was  rated  as  being  the 
most  important,  four  points  for  the  next  most  important,  etc. 

Clinicians  Directors  Evaluators 

Number  Points  Number  Points  Number  Points 


5 

139 

5 

55 

5 

39 

1 

134 

1 

49 

1 

27 

2 

107 

4 

43 

2 

27 

4 

103 

2 

38 

3 

24 

3 

82 

3 

34 

4 

23 
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No  clinician  interviewed  stated  that  he  was  "against"  formal 
evaluation.  However,  many  clearly  hold  reservations  about  the 
validity  of  evaluation  technology.  They  see  the  lack  of  resources 
to  develop  an  adequate  technology  of  formal  evaluation  as  an  even 
more  important  obstacle. 

Evaluators  themselves  hold  reservations  about  the  status 
of  the  technology  of  formal  evaluation.  Table  3.  11  shows  that  they 
ranked  the  unreliability  of  evaluation  technology  as  second  only  to 
the  lack  of  resources  as  an  obstacle  to  the  successful  implementation 
of  formal  evaluation.  Because  of  the  non -threatening  nature  of  most 
of  the  evaluations,  the  evaluators  had  not  been  directly  challenged 
as  to  the  validity  of  formal  evaluation.  However,  when  the  author 
asked  one  evaluator  how  he  thought  that  the  staff  would  react  to  a 
negative  evaluation,  he  described  what  he  thought  would  be  the 
most  likely  reaction  as  follows: 

(Evaluator)  Oh,  I think  that  the  first  thing  they  will  say 
is  that  it  is  invalid  and  that  the  methodology  is  all  screwed  up, 
that  it  doesn't  show  anything.  Or,  they  will  point  to  the 
techniques  and  the  methodology  and  the  way  the  evaluation  is 
done  to  discredit  it. 

The  only  negative,  personal  evaluation  observed  by  the  author  was 
when  a particular  evaluator  kept  time  logs  on  the  clinical  staff 
concerning  how  they  spent  their  time.  He  came  up  with  surprisingly 
low  figures  for  the  amount  of  time  the  clinicians  spent  in  direct 
services.  Some  clinicians  did  challenge  his  figures  as  being  invalid. 
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The  following  director  explains  why  he  feels  that  his  organization 
should  not  be  directly  compared  to  another  mental  health  organization 
in  an  adjacent  county: 

(Director)  To  judge  the  effectiveness  of  a community 
program,  let's  take  for  example  reducing  the  number  of 
admissions  to  a state  mental  hospital  as  a measure  of 
effectiveness,  I think  that  that  is  a simple-minded  measure 
of  . . . effectiveness.  Take,  for  example,  [ name  of  a mental 
health  organization  in  an  adjacent  counts].  It  has  federal 
funds  for  staffing  and  can  take  care  of  people  better  locally 
because  they  have  the  money  to  do  it.  To  compare  our  admission 
rate  with  a community  that  is  better  able  financially  to  take 
care  of  this  kind  of  patient  is  an  unfair  comparison.  And  it 
just  scares  me  that  the  people  juggling  these  figures  don't 
know  the  differences  in  the  communities. 

At  the  program  level,  one  supervisory  clinician  in  a rural 

area  argued  that  figures  such  as  those  concerning  time  spent  in 

delivering  direct  service  did  not  have  the  same  meaning  as  they  did 

in  a more  urbanized  area: 

(Supervisory  Clinician)  You  could  compare  this  area 
with  other  areas.  Sure,  for  example,  you  could  tell  whether 
or  not  the  patients  are  attaining  their  goals  as  well  as  they 
are  in  other  areas.  O.K. , but  what  decisions  do  you  make 
from  this  ? I think  that  you  have  to  look  at  what  the  constraints 
are.  Here,  for  example,  it  is  very  hard  for  patients  to  get 
in  to  see  you  which  is  why  I didn't  feel  bad  about  spending 
only  15  percent  of  my  time  in  direct  service. 

Some  clinicians  reject  even  the  most  rigorous  of  evaluation 

methodologies  such  as  experimental  models: 

(Clinician)  I can  only  give  you  my  own  biased  opinion 
and  that  is,  that  in  my  concept  of  a human  being,  there  are 
so  many  complications,  so  many  things  that  can  affect  what 
we  do,  that  I don't  see  how  you  can  have  a control  that  even 
half-way  covers  a few  of  these  factors.  So,  no,  I don't 
see  any  purpose  in  having  controls  at  all. 
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The  author  asked  the  same  clinician  if  she  though  that  formal 
evaluation  could  be  used  to  compare  the  benefits  of  different  programs 

(Clinician)  Nobody  seems  to  have  devised  any  particular 
methodology  of  any  kind  that  does  any  good.  So  how  in  the 
world  can  they  decide  whether  this  kind  of  therapy  does 
better  than  that  kind? 

Nevertheless,  the  clinician  mentioned  above  rated  the  performance 
of  the  evaluator  in  the  organization  quite  favorably  because  the  latter 
had  helped  her  write  grant  proposals  and  had  not  attempted  to  make 
invidious  comparisons  between  programs. 

To  summarize,  the  attitudes  held  by  clinicians  and  directors 
in  the  organization  are  ambiguous.  They  generally  support  high'iv 
generalized  evaluations  but  retain  reservations  about  the  validity 
of  current  evaluative  techniques.  There  was  little  evidence  that 
skepticism  concerning  evaluative  techniques  has  led  to  direct 
opposition  to  the  implementation  of  any  evaluation.  However,  the 
lack  of  such  resistance  was  the  consequence  of  the  non-threatening 
nature  of  current  evaluations  more  than  the  result  of  confidence  in 
evaluation  technology.  Many  clinicians  and  directors  feel  strongly 
that  no  formal  evaluation  can  determine  the  comparative  effectiveness 
of  organizations  and  programs.  Most  clinicians  and  directors 
believe  that  the  particular  constraints  of  their  programs  and 
organizations  must  be  understood  before  any  evaluative  judgments 


are  made. 
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Thus,  it  would  appear  that  weaknesses  in  methodologies  of 

evaluation  and  comparisons  based  on  formal  evaluations  can  always 

be  found.  As  a consequence,  clinicians  and  directors  are  likely 

to  be  able  to  find  a basis  to  reject  an  evaluation  which  disagrees 

with  their  basic  beliefs.  Thus  methodological  weaknesses  are 

likely  to  turn  the  debates  raised  by  evaluations  into  a struggle  among 

9 

individuals  with  different  value  systems  or  subjective  preferences. 
When  this  occurs,  the  outcome  is  more  likely  to  be  determined 
by  the  relative  power  held  by  the  participants  in  such  debates 
rather  than  by  the  force  of  evidence  from  scientific  evaluations. 
Despite  the  further  development  of  the  technology  of  mental  health 
evaluation,  this  same  phenomenon  of  technical  evaluative  issues 
being  reduced  to  questions  of  values  and  power  is  likely  to  occur. 


Alternative  Forms  of  Evaluation 
In  the  first  section  of  this  chapter,  the  attitudes  of  key  groups 
of  personnel  in  these  organizations  were  examined  with  particular 
focus  given  to  the  degree  of  anxiety  aroused  by  formal  evaluation 
and  skepticism  concerning  its  technology.  However,  it  has  been 


®Thus  James  Q.  Wilson  argues  that  evaluators  will  find  policy 
interventions  effective  or  non-effective  depending  on  whether  the 
researchers  evaluating  the  impact  support  or  oppose  the  policy. 

See  James  Q.  Wilson,  "On  Pettigrew  and  Armor:  An  Afterword,  " 
The  Public  Interest,  No.  30  (Winter,  1973),  pp.  132-134.  See 
also  the  discussion  concerning  this  point  by  Walter  Williams,  Social 
Policy  Research  and  Analysis  (New  York:  American  Elsevier,  1971), 
p.  104  and  p.  123. 
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emphasized  that  formalized  evaluations  did  not  play  a significant 
role  in  organizations  decision  making,  whether  important  or  routine 
decisions  were  involved. 

Nevertheless,  decisions  are  made  daily  which  require  evaluative 
judgments  in  choosing  between  alternative  courses  of  action. 

Examples  of  the  kinds  of  decisions  being  referred  to  include  such 
issues  as  determining  who  is  doing  a good  job  in  the  organization, 
which  programs  should  receive  a higher  percentage  of  revenues, 
and  how  much  emphasis  should  be  placed  on  various  kinds  of  programs. 
Since  formal  evaluation  did  not  have  much,  if  any,  impact  on  the 
decision  making  process,  the  kinds  of  evaluation  actually  used  to 
reach  such  decisions  are  examined  here. 

A primary  form  of  evaluation  is  the  openly  subjective  and 
intuitive  type  of  judgment  used  by  many  clinicians  and  directors 
to  evaluate  the  work  of  their  subordinates  or  colleagues.  While  it 
may  be  supplemented  by  other  data,  the  directors  and  clinicians 
interviewed  were  quite  frank  about  the  subjective  basis  of  most  of 
their  judgments.  Moreover,  many  of  these  directors  and  clinicians 
feel  that  the  subjective  form  of  evaluation  is  quite  valid  as  revealed 
in  the  following  comments: 

(Director  #1)  I know  for  a fact  that  there  are  some 
kinds  of  therapists  that  work  fantastically  with  one  kind  of 
patient  but  blow  it  with  another  kind.  Maybe  I am  being 
defensive,  I don't  know,  but  I do  think  that  the  gut  feeling 
type  of  evaluation  that  therapists  do  of  one  another  has  an 


98 


awful  lot  of  validity,  and  I wonder  if  we  subjected  them  to  a 
microscopic  kind  of  scrutiny,  that  that  wouldn't  kill  some- 
thing that  goes  on  between  therapist  and  the  patient. 

(Director  #2)  I myself  don't  feel  that  the  state  of  the 
art  of  evaluation  is  very  well  developed.  Then,  again,  you 
know,  when  you  work  with  people,  you  know  (although  I 
know  that  I can't  prove  it  with  any  kind  of  validity)  what  the 
place  does.  I know  what  the  weak  points  are  and  I know  what 
the  strengths  are.  And,  subjectively,  I know  pretty  much 
who  the  strong  therapists  are,  and  who  the  weak  ones  are. 

(Supervisory  Clinician)  A lot  of  people  would  disagree 
with  me  on  this  but  I think  that  the  quality  of  a person's  treat- 
ment can  be  easily  evaluated.  You  can  watch  the  way  they 
handle  themselves  as  they  progress  along  the  line  from 
admittance  to  discharge.  I have  a conceptualization  of  the 
way  that  I think  that  treatment  should  be  and  I tell  or  advise 
others .... 

As  would  be  expected,  evaluators  took  issue  with  the  argu- 
ment that  such  subjective  evaluations  can  be  valid: 

(Evaluator  #1)  I think  that  any  evaluation  is  going  to  be 
subjective.  But  for  an  individual  to  say  that  he  can  subjectively 
but  validly  evaluate- -I  think  that  is  bullshit.  I think  we  wind 
up  with  degrees  of  subjectivity.  . . . 

(Evaluator  #2)  There  are  people  around  here,  of  course, 
that  I don't  like,  but  that  does  not  mean  that  they  are  not 
good  clinicians,  or  that  they  are  not  good  in  the  position  that 
they  are  in.  So  I guess  that  I would  tend  to  rely  more  on  a 
formal  type  of  evaluation. 

Several  clinicians  and  some  directors  agree  that  the  validity 
of  these  subjective  evaluations  was  questionable.  One  director 
argues  that  such  judgments  are  self-reinforcing: 

(Director)  If  I have  an  idea  that  massage  therapy  or 
individual  therapy  is  a good  thing,  and  I do  it  and  I get  paid 
for  it,  then  with  the  reinforcement  schedule  that  I have  set 
up  for  myself,  I can  convince  myself  that  such  and  such  method 
is  a really  good  thing.  You  just  look  at  these  people  and  they 
are  better  period.  That  is  the  clinical  impression. 
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A clinician  argues  that  subjective  evaluations  are  particularly 
difficult  when  the  "evaluator"  and  the  person  being  evaluated 
disagree  on  values  or  are  separated  by  cultural  differences: 

(Supervisory  Clinician)  Subjective  evaluation  is  very 
tricky- -that  is  doing  things  by  instincts.  We  love  to  talk. 

That  is  one  of  the  most  enjoyable  things  that  people  do-- 
talking  to  one  another  about  somebody  else.  But  the  whole 
social  service  structure  is  basically  involved  in  subjective 
evaluation.  . . Where  you  really  get  in  trouble  is  when  you 
have  cross-cultural  differences. 

However,  despite  the  doubts  which  many  clinicians  and  directors 
have  about  intuitive  evaluations,  there  was  general  agreement 
among  those  interviewed  that  many  day-by-day  decisions  are  made 
on  this  basis.  Moreover,  the  confident  manner  in  which  such 
intuitive  evaluations  are  used  contrasts  sharply  with  the  hesitant 
manner  in  which  formalized  evaluations  are  employed  in  these  same 
organizations. 

Most  decision  makers  in  these  organizations  do  seek  out 
additional  information  when  it  is  available  to  supplement  their 
subjective  impressions.  The  following  director  describes  how 
evaluators  had  collected  data  which  contradicted  his  impressions 
about  the  comparison  between  the  effectiveness  of  two  different 
services.  However,  he  concludes  that  he  is  right  about  which  is  the 
better  service  despite  the  formal  data  presented  by  the  evaluators; 

(Director)  The  evaluators  have  given  me  something 
that  shows,  for  instance,  that  one  of  the  services  that  we 
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have  at  the  county  hospital  has  an  average  of  12.  5 days  of 
hospitalization  which  is  worse  than  the  other  which  has  a 
10  days  average.  But  the  one  with  12.  5 average  has  only  24.  6 
percent  going  to  the  State  Hospital  whereas  the  one  with  a 
10  days  average  has  41.  5 percent  going.  Well,  this  might  be 
significant  but  there  are  so  many  other  things  involved  in 
this  that  it  is  a little  difficult  to  make  that  conclusion.  The 
one  with  a 10  day  average  I thought  would  be  the  longer  one. 

And  the  one  that  is  longer,  12.5  days,  which  is  a little  bit 
longer  than  we  are  shooting  for  but  which  I feel  is  the  better 
service.  . . . 

A supervisory  clinician  describes  how  he  uses  a wide  variety 
of  data  to  evaluate  the  therapists  under  his  direction  with  a combina- 
tion of  tapes,  statistical  figures,  and  an  assessment  of  the  groups 
or  clientele  they  are  working  with: 

(Supervisory  Clinician)  We  do  it  [evaluation]  in  a lot 
of  ways.  I do  it  a lot  by  taping  them  and  listening  back  over 
their  sessions,  and  see  how  they  are  handling  the  patient.  I 
also  look  at  the  dropout  rates.  For  example,  if  they  have  twelve 
in  a group  and  after  three  sessions,  they  are  down  to  three, 

I am  going  to  jump  in  there.  With  a good  evaluation,  you 
might  look  at  the  people  and  say,  "Well,  maybe  this  was 
a bad  group  to  start  with.  " Or  you  may  look  at  the  therapist 
and  decide  that  his  method  of  dealing  with  alcoholic  people 
is  not  as  good  as  his  method  of  dealing  with  the  children  of 
alcoholics.  It  is  difficult.  I don't  know  that  anyone  has  ever 
really  gone  into  it.  I see  it  as  an  interesting  dissertation  for 
somebody  to  do.  How  do  you  evaluate  professionals? 

Thus  a broad  and  eclectic  selection  of  information  is  combined 

with  a supervisor's  impressions  to  reach  judgments  concerning  the 

performance  of  individual  programs  or  therapists  in  the  above  cases. 

While  formal  evaluation  such  as  goal  attainment  scores  might  be 

used  as  an  additional  source  of  information,  it  is  difficult  to  see 

how  any  formal  evaluation  procedure  can  replace  the  flexibility  and 
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openness  of  the  above  kind  of  reasoning. 

Of  course,  the  flexibility  and  discretion  that  enable  some 
supervisors  to  make  highly  perceptive  judgments  in  some  cases  can 
also  lead  to  arbitrary  and  unfair  evaluations  in  other  cases.  Indeed, 
some  supervisory  clinicians  seem  to  have  developed  some  rather 
questionable  yardsticks  for  deciding  which  of  their  subordinates  is 
doing  a good  job.  The  following  clinician  evaluates  his  subordinates 
on  the  basis  of  who  comes  to  him  for  advise  the  least: 

(Supervisory  Clinician)  I have  heard  other  people  saying 
the  same  thing:  the  person  who  is  doing  the  best  job  comes 
to  you  least.  That  is,  both  they  and  their  patients  come  to 
you  with  the  fewest  problems.  I tend  to  think  of  them  as 
doing  the  best  job.  That  doesn't  hold  right  down  the  line. 

Some  of  them  may  come  to  you  for  a lot  of  reassurance,  to  see 
if  they  are  doing  a good  job.  But  that  is  different  from  them 
actually  goofing  up,  the  patients  getting  into  trouble  a lot.  . . . 

As  the  statement  above  suggests,  much  of  the  success  of  the 

therapy  depends  on  the  client  rather  than  the  therapist.  Subjective 

evaluations  can  take  into  consideration  the  role  that  the  client  plays 

in  the  success  of  the  therapy.  Indeed,  one  supervisory  clinician 

argues  that  there  would  be  more  value  to  an  evaluation  of  the  clients 

than  an  evaluation  of  the  therapists.  Likewise,  he  states  that  clients 

should  be  taught  how  to  get  more  out  of  their  therapy: 

(Supervisory  Clinician)  Maybe  the  patient  should  see  the 
data  on  his  charts  and  maybe  the  patient  should  be  evaluated. 

If  you  want  to  do  research,  do  it  on  the  patients  and  question 
them  on  how  they  can  make  the  most  of  their  opportunities. 

So,  what  I am  saying  is,  that  maybe  we  are  doing  research 
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from  the  wrong  end- -focusing  in  on  the  therapist.  And,  by- 
God,  we  really  ought  to  be  looking  at  it  from  the  consumer's 
point  of  view.  ^ 

Based  on  participant  observation,  it  appears  that  most 
important  decisions  rely  to  a great  extent  upon  subjective  preferences 
and  values  held  by  the  decision  makers.  Objective  data  and  formal 
evaluations  maybe  utilized  in  the  making  of  such  decisions,  but  they 
do  not  appear  to  determine  the  final  judgment  made.  The  objective 
data  needed  to  make  the  specific  decision  maybe  unavailable.  Even 
if  such  data  are  available,  it  is  often  ambiguous  concerning  the 
implications  to  be  drawn. 

A good  example  of  the  above  points  occurred  in  an  organization 
in  which  there  had  been  a long-standing  debate  about  the  value  of 
an  experimental  program.  This  program  used  paraprofessionals, 
who  are  paid  substantially  less  than  professionals.  Both  the 
evaluator  and  director  in  this  organization  were  skeptical  of  the 
abilities  and  performance  of  these  paraprofessionals  and  desired 
to  replace  them  with  professionals  such  as  psychiatrists.  When 
several  paraprofessionals  left  the  organization,  the  director  did  not 
immediately  fill  their  positions  with  other  paraprofessionals  but 
hoped  to  combine  the  salaries  of  several  of  these  paraprofessionals. 

^In  fact,  this  clinician  actually  went  to  the  evaluator  and 
suggested  that  they  begin  a program  like  the  one  he  describes  in  his 
comments  here  after  the  conclusion  of  this  interview. 
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in  order  to  hire  a single  professional. 

However,  many  of  the  clinicians  who  were  the  supervisors  of 
these  paraprofessionals  felt  that  they  were  doing  an  adequate  or 
very  good  job.  In  fact,  some  preferred  them  to  professionals 
because  the  paraprofessionals  knew  the  communities  better  and 
spent  more  of  their  time  in  direct  service.  Many  of  the  pro- 
fessionals served  as  consultants,  moving  from  area  to  area.  Such 
was  the  situation  when  the  following  exchange  took  place  at  a 
meeting  when  the  evaluator  and  the  director  hoped  to  convince  the 
others  to  hire  more  professionals: 

(Evaluator)  We  need  to  take  a look  at  the  staffing  pattern. 
This  is  a major  question. 

(Supervisory  Clinician  #1)  We  need  to  plan  for  a train- 
ing program  for  the  new  paraprofessionals  eventually. 

(Evaluator)  Would  a paraprofessional  be  more  practical 
than  a professional? 

(Supervisory  Clinician  #1)  We  need  both  but  I would 
prefer  one  paraprofessional. 

(Director)  A professional  could  generate  money. 

(Supervisory  Clinician  #1)  I don't  see  [name  of  a 
psychiatrist  in  the  organization]  generating  money.  He  could 
but.  . . . Sometimes  we  have  more  professional  consultants 
in  this  area  than  people  involved  in  direct  service.  And 
the  patient  care  falls  on  my  staff  of  paraprofessionals. 

(Director)  How  effective  do  you  find  the  paraprofessionals 
are  in  meeting  the  direct  services?  What  is  the  optimum  mix 
of  the  staff? 

(Supervisory  Clinician  #2)  You  can't  compare  them  to  do 
the  same  things  as  professionals --if  you  are  going  to  try  to 
evaluate  them. 
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(Supervisory  Clinician  #3)  In  my  area,  you  have  chronic 
problems  and  don't  need  any  great  psychotherapeutic  skills.  . . . 
Great  intellectual  problems  don't  exist. 

(Supervisory  Clinician  #4)  I would  say  that,  in  general, 
two  paraprofessionals  can  do  more  than  one  professional. 

(Director)  But  we  don't  have  a good  outcome  study. 

For  example,  if  one  professional  and  two  paraprofessionals 
take  in  five  intakes,  could  that  be  done  in  a better  way? 

To  carry  out  a good  outcome  study  to  determine  the  optimum 
staff  ratio  would  take  a great  deal  of  time  and  resources  and 
probably  would  still  be  open  to  questions  of  methodological  weak- 
nesses. More  importantly,  such  questions  as  those  above  face 
the  decision  makers  of  these  organizations  daily.  Decisions  have 
to  be  made  before  rigorous  studies  can  provide  complete  information. 
Directors,  therefore,  decide  on  the  basis  of  preferences  and  what- 
ever information  happens  to  be  available.  Evaluative  techniques 
such  as  goal  attainment  scaling,  consumer  satisfaction  surveys, 
needs  assessment  surveys,  and  problem-oriented  records  are  not 
designed  to  provide  answers  to  specific  organizational  decisions 
such  as  how  many  paraprofessionals  do  work  equal  to  that  of 
professionals. 

Decisions  such  as  how  to  invest  extra  money  also  appear  to 
be  determined  by  a large  variety  of  sources  of  information  such  as 
the  waiting  lists  of  different  programs  and  feedback  from  the 
community  as  one  director  describes  the  process: 
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(Director)  Well,  when  the  time  comes  and  that  happens 
[additional  money  becomes  available],  you  have  a lot  of 
things  that  help  you  decide.  If  you  have  good  contacts  with 
the  community,  they  let  you  know  what  their  needs  are. 

For  example,  they  call  you  and  say:  "I  want  this  service  and 
no  one  in  the  County  has  it  and  it  seems  to  me  to  be  in  your 
province.  Why  don't  you  have  it  there?"  Another  thing  is 
the  waiting  list.  If  youth  services  has  such  and  such  a waiting 
list,  you  start  looking  at  what  the  community's  needs  are  for 
youth  services  but  also  at  how  it  is  utilizing  its  people.  Are 
they  doing  a lot  of  consultation  or  more  direct  service?  So 
I would  have  to  base  it  on  several  things.  And  I am  sure  that 
ten  other  factors  would  come  into  it  that  I can't  even  think  of 
right  now. 

This  decisional  process  is  similar  to  that  found  by  Nienaber  and 
Wildavsky  in  recreation  agencies  (rather  than  the  type  which  was 
supposed  to  prevail): 

Evaluation  is  done  ad  hoc,  in  response  to  changing 
park  conditions.  It  is  surely  not  the  type  of  analysis  that 
will  lead  to  radical  changes  in  the  agency's  performance 
of  goals.  It  does  not  question  the  fundamental  assumptions 
of  the  organization,  but  it  does  respond  to  immediate  problems 
with  which  Park  personnel  have  to  contend.  ^ 

Unless  evaluations  are  designed  for  specific  decisions,  they 

are  likely  to  be  very  inadequate  sources  of  information  for  making 

choices  among  alternative  courses  of  action.  Furthermore,  simple 

data  such  as  the  figures  on  waiting  lists  or  advice  from  significant 

members  of  the  community  will  often  be  more  influential  than,  for 


■^Jeanne  Nienaber  and  Aaron  Wildavsky,  The  Budgeting  and 
Evaluation  of  Federal  Recreation  Programs  or  Money  Doesn't 
Grow  on  Trees  (New  York:  Basic  Books,  1973),  p.  53. 
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example,  an  epidemiological  survey.  Methodologically,  simple 
evaluations  are  advocated  by  several  directors  and  clinicians  as 
indicated  by  the  following  comments: 

(Director)  How  do  you  evaluate?  There  are  so  many 
obvious  simple  kinds  of  evaluations  that  never  get  thought  of. 

For  instance,  a Center  such  as  this  one,  which  has  been 
designated  a bad  Center,  will  change,  for  example,  by 
hiring  a sociological -anthropological  type  to  interview  every- 
body--the  board,  the  staff  and  to  develop  a case  history  of 
what  the  community  had  developed  in  the  way  of  service. 

(Clinician)  That  is  really  exciting  to  me.  It  seems  to 
me  that  what  I am  trying  to  say  is  that  if  we  can  get  back  to 
common  sense,  although  I guess  if  you  believe  in  common 
sense  you  also  believe  the  world  is  flat,  but  there  are  a lot 
of  simple  things.  You  don't  need  a fancy  evaluation.  . . but 
simple  kinds  of  questions  with  simple,  honest  answers.  If 
you  follow  the  statistics  in  psychology,  the  more  polish  the 
article  has  statistically,  the  less  use  it  is.  Simplicity  in 
research  is  essential.  If  you  get  to  the  level  where  staff 
members,  clients,  et  cetera,  can  all  understand,  then  it 
is  useful. 

In  summary,  both  participant  observation  and  interviews  indicated 
that  formal  evaluations  are  likely  to  play  only  a very  small  part 
in  the  making  of  most  of  the  decisions  in  these  organizations.  Formal 
evaluation  is  only  one  of  many  kinds  of  data  that  form  the  basis  of 
important  decisions.  Expectations  of  more  than  incremental  change 
from  formal  evaluation  are  likely  to  be  frustrated. 

Some  directors  attempt  to  develop  certain  standards  or  ad 
hoc  models  against  which  to  compare  the  performance  of  the  personnel 
in  the  organizations.  One  director  used  industrial  organizations  as 
his  frame  of  reference  in  determining  the  efficiency  of  the  persons 
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under  his  supervision.  He  argued  that  in  industry  five-eighths 
is  an  efficiency  criterion  which  the  mental  health  organization 
should  adopt  as  a goal  for  the  fraction  of  total  time  that  should  be 
spent  in  direct  service.  The  following  situation  occurred  when  the 
evaluator  collected  data  which  showed  that  an  average  of  only  about 
15  percent  of  the  clinicians'  time  was  being  devoted  to  direct  service, 
far  below  the  director's  efficiency  criterion  of  62.  5 percent.  The 
director  expressed  dismay  at  the  figures,  but  other  clinicians  in 
the  organization  were  far  less  shocked: 

(Supervisory  Clinician  #1)  What  is  the  standard  against 
which  we  are  measuring.  . . ? 

(Director)  In  some  places  twenty-five  of  forty  hours 
is  the  standard.  . . [cites  a study].  . . and  also  industry 
and  private  practice.  The  only  thing  that  counts  is  how  much 
service,  direct  and  indirect,  that  we  are  providing. 

(Supervisory  Clinician  #1)  I think  that  it  should  be  how  the 
people  feel  about  us  and  the  service  we  are  providing,  not 
the  number  of  hours.  . . . 

(Director)  Whenever  you  go  into  a place  that  hasn't 
had  service  before,  they  will  think  that  you  are  great  [the 
above  supervisory  clinician  had  responsibility  for  an  area 

that  had  not  had  mental  health  services  previously] 

Right  now  you  are  far  below  the  60  percent  level  and  that  is 
an  administrative  question. 

(Supervisory  Clinician  #1)  I don't  think  that  really 
reflects  the  direct  service.  . . . People  don't  come  into 
see  you  twenty-five  hours  a week.  . . [goes  on  to  say  that 
he  spends  much  of  his  time  on  the  phone  with  clients  ] . . . . 

(Evaluator)  But  you  don't  get  paid  for  that.  You  only 
get  paid  a fee  for  direct  service. 

(Supervisory  Clinician  #2)  My  whole  concern  is  different-- 
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a philosophical  one.  I am  interested  in  providing  indirect 
service  and  worried  about  putting  too  much  emphasis  on 
direct  service. 

(Director)  I don't  think,  that  most  of  your  paraprofessionals 
[who  were  under  the  supervision  of  the  above  clinician]  are 
capable  of  providing  indirect,  consultee  service.  I really 
don't  see  that  as  a defensible  figure--15  percent.  If  it  were 
40  percent,  you  could  defend  it,  but  it  is  embarrassingly 
low. 

The  above  discussion  illustrates  the  point  that  all  data  are  potentially 
ambiguous.  Also,  skepticism  concerning  the  technology  of  formal 
evaluation  has  been  emphasized  previously.  However,  it  is  evident 
from  the  above  discussion  that  more  subjective  evaluations  can 
arouse  the  same  kind  of  skepticism  and  accusations  of  bias. 

All  evaluations,  both  formal  and  informal,  are  inextricably 
tied  up  in  values.  Thus  evaluations  are  likely  to  arouse  conflict 
unless  there  is  an  unusually  strong  consensus  concerning  values 
that  should  be  emphasized.  The  ad  hoc  models  used  by  different 
directors  and  clinicians  to  judge  the  efficiency  and  effectiveness 
of  their  organizations  and  programs  thus  revealed  much  about  their 
value  systems.  Some  directors  emphasized  a "private  practice 
model"  as  the  criterion  against  which  the  efforts  of  mental  health 
organizations  should  be  measured.  Others  disagreed  with  the 
analogy  between  private  practice  and  public  service: 

(Director)  If  we  can't  do  things  as  well  as  if  we  were 
in  private  practice,  then  maybe  we  should  disband  the  organi- 
zation. That  is,  if  it  costs  more  than  twenty -five  dollars  an 
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hour,  we  should  close  down  the  place  and  have  secretaries 
refer  clients  to  private  therapists. 

(Child  Psychologist)  There  is  at  least  one  fallacy--the 
clientele.  It  costs  more  to  deliver  services  to  poor  people. 

The  needs  are  different. 

(Director)  I am  not  suggesting  that  is  what  we  should  do, 
but  that  is  something  against  which  to  measure  our  efforts. 

(Child  Psychologist)  I can  gauge  how  much  I do  by  how 
much  paper  and  client  contact  I have. 

A very  different  kind  of  ad  hoc  model  was  developed  at  another 
organization  where  evaluations  that  focused  on  quantitative  figures 
or  analogies  to  industry  were  rejected  in  favor  of  an  evaluation 
that  was  derived  from  the  philosophy  of  humanistic  self-actualiza- 
tion. The  director  of  the  organization  argued  that  the  only  goal  of 
the  organization  was  to  carry  on  non -exploitative  relationships 
among  themselves,  with  the  clients  of  the  organization,  and  with 
the  total  community.  He  stated  that  they  completely  rejected  any 
technological  model  of  evaluation.  He  indicated  that  the  community 
would  evaluate  the  work  of  the  organization  positively  when  they 
became  aware  that  the  organization  was  an  "energy  source"  for  the 
whole  community. 

However,  while  the  philosophy  of  the  organizational  members 
rejected  the  idea  of  using  numbers  to  evaluate  its  programs, 
nevertheless,  the  director  kept  several  large  graphs  in  his  office 
that  sought  to  demonstrate  that  the  efficiency  of  the  organization 
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had  increased  since  they  had  adopted  the  philosophy  of  self- 
actualization.  Also,  the  organization  was  continuing  to  collect  and 
send  to  the  agencies  which  funded  them  the  numerous  statistics 
required.  The  director  admitted  that  he  was  compromising  with 
the  external  culture  and  its  evaluative  standards  even  though  he 
totally  disagreed  with  them.  In  fact,  it  seemed  clear  that  the 
members  of  this  organization  were  determined  to  succeed  in  terms 
of  the  quantitative  and  technological  model  of  evaluation  in  order  to 
prove  the  worth  of  their  philosophy  to  "outsiders.  " While  this 
illustration  strongly  suggests  the  importance  of  the  organizational 
struggle  for  survival,  and  also  has  implications  for  the  issue  of 
accountability  (both  of  which  will  be  explored  in  later  chapters), 
here  it  demonstrates  how  all  evaluations  are  inextricably  intertwined 
in  the  value  systems  of  the  evaluators.  However,  because  of  their 
openness,  formal  evaluations  are  likely  to  make  the  values  which 
they  emphasize  more  clear  than  informal  evaluations. 

The  fact  that  values  influence  evaluations  creates  the  potential 
for  continued  conflict  in  organizations  where  there  exist  sharp 
differences  in  values  held  by  different  organizational  members. 

In  one  organization,  there  tended  to  be  a division  into  two  groups 
that  took  opposing  stands  on  several  issues  including  the  follow- 
ing: 


Ill 


Group  One 

(1)  favored  emphasis  on  direct 
services 


(1)  favored  emphasis  on 
indirect  services 


Group  Two 


(2)  favored  centralization  in  the 
organization 


(2)  favored  decentralization 
in  the  organization 


(3)  favored  emphasis  on  poor, 
schizophrenic  type  of  client 


(3)  favored  emphasis  on 
middle  class,  neurotic 
type  of  client 


(4)  favored  a higher  percentage  of 
professionals  on  the  staff 


(4)  favored  retention  of  the 
same  percentage  of  para- 
professionals  on  the  staff 


Whenever  an  issue  was  discussed,  each  side  would  present  the 
evidence  in  favor  of  its  position.  For  example,  the  first  group 
would  emphasize  the  paraprofessionals ' inability  to  handle 
individuals  with  complex  problems,  while  the  second  group  would 
cite  the  advantages  of  their  knowledge  of  the  community. 

Sometimes  objective  data  are  entered  into  the  debate,  as  when 
a psychiatrist  with  supervisory  responsibilities  examined  the 
activity  sheets  of  a sample  of  paraprofessionals  and  found  that  they 
had  seen  an  average  of  only  one -and -a -half  clients  a day.  Such 
data  are  not  unambiguous , however,  because  others  replied  that  the 
efficiency  of  the  paraprofessionals  would  increase  in  the  future, 
or  that  they  spent  much  of  their  time  on  the  telephone  with  clients 
rather  than  seeing  them  directly.  That  the  same  data  could  lead 
to  different  decisions  was  pointed  out  by  a supervisory  clinician 
in  another  organization  when  he  was  asked  if  formal  evaluation  could 
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be  used  to  determine  the  internal  allocation  of  resources  to  different 
programs : 


(Supervisory  Clinician)  The  more  I think  about  your 
question,  the  more  merit  I think  it  has --the  individual 
program  being  evaluated  in  order  to  compare  its  efficiency 
■with  that  of  another  program.  There  is  a kind  of  intriguing 
element  in  it.  I don't  know  what  more  to  say  about  that  but 
I think  that  there  are  certain  phases  of  our  operation  here 
which  are  more  efficient  than  others.  If  we  did  find  this,  we 
might  not  allocate  more  to  the  one  that  is  more  efficient  but 
to  the  one  that  is  less  efficient,  so  they  can  get  better. 

The  ultimate  ambiguity  of  all  objective  data  leaves  the  final 

decisions  to  be  based  on  subjective  judgments.  When  no  consensus 

exists  as  to  what  these  judgments  should  be,  as  in  some  of  the 

cases  cited  above,  the  principle  of  hierarchy  comes  into  play  as 

the  common  organizational  form  of  resolving  such  impasses. 

Thus  the  growth  of  formal,  scientific  evaluation  is  more  likely  to 

become  part  of  the  power  struggle  in  organizations  than  to  replace 


Summary 

This  chapter  explored  the  attitudes  held  by  evaluators, 
clinicians,  and  directors  toward  formal  evaluation.  Attitudes 
toward  informal  evaluations  were  also  examined.  Data  from 
questionnaires,  participant  observation,  and  interviews  show  that 
most  clinicians  strongly  favor  generalized  evaluations  which  focus 
on  the  total  organization  or  programs.  Most  clinicians  were  much 
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less  enthusiastic  or  even  opposed  to  the  use  of  formal  evaluations 
to  judge  the  work  of  individual  therapists. 

Evaluators  are  no  more  favorable  toward  carrying  out 
threatening  evaluations  than  are  the  clinicians.  Based  on  inter- 
views and  participant  observation,  this  reluctance  of  the  evaluators 
would  seem  to  be  due  in  large  part  to  the  unpleasantness  they 
anticipated  if  they  were  to  undertake  such  evaluations  of  co-workers. 
Thus,  there  was  little  evidence  that  the  unprogrammed  norms  of  the 
informal  organization  were  directly  responsible  for  the  lack  of 
impact  of  formal  evaluations  introduced  in  these  organizations. 

There  did  not  appear  to  be  any  active  resistance  to  the  evaluator. 
Rather,  the  effect  of  such  informal  elements  was  indirect:  the 
evaluator  anticipating  resistance  and  attempting  to  forestall  it  by 
following  a non -threatening  way  of  introducing  evaluations. 

The  clinicians  rated  skepticism  concerning  the  technology  of 
formal  evaluation  as  the  second  greatest  obstacle  to  its  successful 
implementation.  Even  the  evaluators  recognized  that  evaluation 
technology  was  open  to  question.  Yet,  the  kind  of  evaluation  which 
all  groups  preferred,  program  evaluation,  is  presently  much  less 
developed  technologically  than  evaluations  of  the  individual  therapist. 
This  inconsistency  suggests  that  attitudes  toward  technology  are  not 
based  exclusively  on  an  objective  assessment  but  are  influenced  by 
other  considerations,  such  as  anxiety. 
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In  the  introductory  chapter,  it  was  noted  that  inductive 
hypotheses  would  be  developed  to  enable  future  testing.  The  above 
arguments  concerning  the  preferences  of  both  clinicians  and 
evaluators  for  non -threatening  evaluations  may  be  summarized 
by  the  following  propositions: 

(1)  Persons  being  evaluated  will  favor  those  evaluations 
which  minimize  potential  threats  to  themselves  and  maximize 
potential  benefits. 

(2)  The  degree  to  which  evaluators  favor  non-threatening 
evaluations  will  be  a positive  function  of  the  amount  of 
resistance  anticipated  by  the  evaluator  and  a negative  function 
of  the  amount  of  authority  possessed  by  the  evaluator. 

The  biggest  obstacle  to  the  acceptance  of  evaluation  technology 
is  a prevalent  view  that  no  two  programs  can  be  perfectly  compared 
through  formal  evaluation.  Some  differentiative  factor  can  always 
be  found  which  tends  to  reduce  such  comparisons  to  preferences 
and  values.  Since  most  decisions  involve  such  comparisons,  this 
theme  of  "non-comparability"  is  seen  as  a potentially  major 
limitation  on  the  kind  of  impact  that  formal  evaluation  can  have  on 
the  decisions  made  in  these  organizations. 

Despite  the  existence  of  some  skepticism  concerning  the 
technology  of  formal  evaluation  and  anxiety  about  the  way  that  such 
evaluations  might  be  used,  it  did  not  appear  that  either  the  technolog- 
ical nor  natural  system  elements  of  the  organizations  offered  a 
good  explanation  as  to  why  evaluation  has  had  such  little  impact  on 
the  decisions  made  by  these  organizations. 
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Some  examples  of  decision  making  were  analyzed  with  parti- 
cular focus  given  to  the  nature  of  the  evaluations  used  to  reach 
these  decisions.  Most  decisions  seem  to  be  made  through  the 
combination  of  a variety  of  data  available  and  the  subjective 
preferences  of  the  decision  maker.  Many  clinicians  and  directors 
feel  strong  confidence  in  their  subjective  evaluations  while  evaluators, 
other  clinicians,  and  directors  strongly  disagree  with  the  validity 
of  these  judgments. 

Yet,  there  seems  to  be  no  way  of  avoiding  the  necessity  of 
relying  upon  such  judgments.  There  is  always  a large  degree  of 
ambiguity  after  formal  evaluation  and  objective  data  have  been  fully 
assembled.  This  ambiguity  is  partly  due  to  the  fact  that  formal 
evaluations  simply  are  not  tailored  to  provide  the  necessary  in- 
formation for  the  numerous  kinds  of  decisions  that  arise  daily. 

Even  if  complete  information  were  available,  ambiguity  can  still 
exist  because  different  interpretations  can  be  drawn  from  the  same 
data  because  of  differing  values  and  perspectives  of  the  specific 
individuals  involved.  It  is  concluded  that  the  introduction  of  formal 
evaluation  is  likely  to  have,  at  best,  an  incremental  effect  on  the 
operations  of  these  organizations. 

Values  are  an  integral  part  of  all  evaluations,  formal  or 
informal.  However,  formal  evaluation  is  more  likely  to  make  clear 
the  kinds  of  values  being  emphasized  by  the  evaluator  than  informal 
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and  subjective  evaluation  is.  When  there  is  a disagreement  as  to 
values  within  these  organizations,  the  introduction  of  formal 
evaluation  can  lead  to  conflict.  There  can  be  just  as  much  skepticism 
and  resistance  to  certain  informal  and  subjective  evaluations. 
However,  decision  makers  are  not  hesitant  to  use  subjective 
evaluations  in  reaching  important  decisions.  By  way  of  contrast, 
formal  evaluations  are  rarely  used  in  such  decisions.  Subjective 
evaluations  are  essentially  the  property  of  the  individuals  making 
them  while  more  formal  and  scientific  evaluations  are  public  and 
open  to  challenge. 

In  this  chapter,  some  of  the  reasons  for  the  lack  of  impact 
of  evaluation  have  been  examined.  Nevertheless,  the  existence 
of  negative  factors  which  prevent  formal  evaluation  from  having 
much  impact  does  not  explain  what  the  positive  forces  were  that 
led  many  of  these  organizations  to  voluntarily  undertake  some  form 


of  formal  evaluation. 


CHAPTER  IV 


THE  FUNCTIONS  OF  EVALUATION 


There  are  several  reasons  to  be  puzzled  as  to  why  the  mental 
health  organizations  under  study  attempted  to  carry  out  formal 
evaluation: 

(1)  The  evaluations  consume  or  are  expected  to  consume 
a significant  amount  of  resources  of  the  mental  health 
organizations. 

(2)  There  is  little  evidence  of  significant  impact  on  policies 
or  programs  from  the  formal  evaluation.  Nor  is  there  an 
expectation  that  such  an  impact  is  likely  to  occur. 

(3)  While  there  is  some  anxiety  about  evaluation  and 
skepticism  concerning  its  technology,  observation  of  the  way 
decisions  are  actually  made  suggests  that  any  formal  and 
scientific  evaluation  is  likely  to  remain  only  a small  part  of 
the  decision  making  process,  even  if  anxiety  and  skepticism 
are  overcome. 

(4)  The  literature  of  organizational  theory  suggests  that 
organizational  survival  and  concern  with  goal  attainment 
are  opposed  "concepts,  " particularly  when  the  organizations 
are  insecure,  as  is  the  case  with  mental  health  centers. 


1See  Alvin  W.  Gouldner,  "Organizational  Analysis,  " in 
Sociology  Today:  Problems  and  Prospects,  ed.  Robert  K.  Merton, 
Leonard  Broom,  and  Leonard  Cottrell  (New  York:  Basic  Books, 
1959),  pp.  400-428. 
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In  this  chapter,  the  positive  functions  and  motivations  which 

led  nearly  all  of  the  fourteen  organizations  to  undertake  some  kind 

of  formal  evaluation  and  several  to  invest  in  a formal  evaluation 

2 

staff  are  analyzed.  First  the  preferences  of  the  clinicians, 
evaluators,  and  directors  are  examined  based  on  questionnaire 
data  concerning  what  they  regard  to  be  the  most  important  functions 
of  evaluation. 

These  preferences  are  then  compared  with  evidence  from 

interviews  and  participant  observation  concerning  the  motivations 

which  led  the  centers  to  invest  in  evaluation.  The  functions  actually 

3 

served  by  formal  evaluation  are  analyzed.  Finally,  differences 
among  the  organizations  in  regard  to  their  involvement  in  evaluation 
are  examined  in  order  to  explain  why  some  organizations  are  very 
interested  in  formal  evaluation  while  others  are  not. 


^All  but  one  of  the  fourteen  organizations  visited  had  either 
undertaken  or  were  planning  to  undertake  some  form  of  formal 
evaluation  as  defined  earlier.  Five  of  the  organizations  had  invested 
in  an  evaluation  staff  at  the  time  of  the  beginning  of  this  study. 

Q 

In  this  chapter,  "function"  is  being  used  to  refer  to  the  actual 
consequences  of  evaluation.  Preferences  expressed  on  question- 
naires may  be  interpreted  as  attitudes  concerning  what  should  be 
the  most  important  consequences  of  evaluation.  See  Robert  Merton, 
Social  Theory  and  Social  Structure  (2nd  ed.  ; Glencoe:  Free  Press, 
1957),  p.  61  and  W.  G.  Runciman,  Social  Science  and  Political  Theory 
(Cambridge:  Cambridge  University  Press,  1965),  pp.  109-134. 
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The  Preferences  of  the  Staff 

It  is  clear  from  the  literature  on  evaluation  that  formal 

5 

evaluation  can  perform  several  functions.  Based  on  a search 
of  the  literature,  a list  of  nine  functions  was  constructed  and  the 
clinicians,  directors,  and  evaluators  were  asked  to  rank  these 

g 

functions  as  to  their  importance.  Table  4. 1 reveals  that  feedback 
and  program  evaluation  are  rated  the  most  important  functions  of 
formal  evaluation  by  the  staff  as  a whole.  Program  evaluation  is 
ranked  as  first  in  importance  by  the  clinicians  and  directors.  The 
evaluators  ranked  program  evaluation  second,  after  feedback,  which 
they  regard  as  the  most  important  purpose  of  evaluation. 


^The  term  "staff"  is  used  to  denote  (unless  noted  otherwise) 
the  composite  of  the  three  major  groups  responding  to  the  question- 
naires: clinicians,  directors,  and  evaluators. 

^See,  e.  g. , Edward  A.  Suchman,  "Action  for  What?  A 
Critique  of  Evaluative  Research,  " in  Evaluating  Action  Programs : 
Readings  in  Social  Action  and  Education,  ed.  Carol  H.  Weiss 
(Boston:  Allyn  and  Bacon,  1972),  pp.  52-84. 

0 

Many  of  these  functions  overlap.  The  evaluation  of  the  quality 
of  programs  would  yield  feedback  to  the  clinical  staff  involved  in 
the  programs.  Thus,  the  differences  among  many  of  the  functions 
is  one  of  emphasis.  However,  all  were  mentioned  often  enough  in 
the  literature  (or  in  participant  observation  and  interviews)  to 
warrant  being  included  in  the  list. 
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TABLE  4. 1 

THE  FUNCTIONS  OF  EVALUATION 


Please  rank  the  following  in  what  you  regard  to  be  their  order 
of  importance  as  functions  of  evaluation.  Evaluation  below  refers 
specifically  to  formal  evaluation  carried  out  by  an  evaluator  and  his 
staff.  (1  = the  most  important  function,  9 = the  least  important  function. ) 

#1  Feedback  for  the  clinical  staff 

#2  Evaluation  of  the  performance  of  individual  staff  members 

#3  Evaluation  of  the  quality  of  the  programs  of  the  mental 

health  organization 

#4  Provide  accountability  to  the  community 

#5  Provide  accountability  to  the  clientele 

#6  Increase  the  efficiency  of  the  organization 

#7  Provide  justification  for  the  funding  of  the  mental 

health  organization 

#8  Determine  the  allocation  of  resources  among  the 

different  service  programs  within  the  organization 

#9  Provide  the  means  to  regulate  the  organizational 

members'  behavior 


The  following  "index  of  importance"  was  constructed  for  each 
group  by  assigning  nine  points  to  the  most  important  function  (as  rated 
by  each  respondent),  eight  points  to  the  next  most  important,  etc.  Thus 
the  following  numbered  functions  with  the  greatest  total  of  points  were 
seen  as  being  the  most  important  functions  by  the  respective  groups. 


Clinicians  Directors  Evaluators 


Number 

Points 

Number 

3 

273 

3 

1 

257 

1 

6 

238 

6 

4 

216 

4 

5 

189 

8 

8 

182 

5 

7 

176 

7 

2 

156 

9 

9 

80 

9 

Points 

Number 

Points 

109 

1 

68 

104 

3 

67 

100 

4 

56 

84 

8 

52 

78 

6 

52 

67 

7 

46 

64 

5 

45 

27 

9 

11 

27 

9 

11 
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While  the  "soft"  nature  of  the  data  warrants  caution  in  their 

interpretation,  particularly  when  only  small  differences  are  involved, 

there  is  a strikingly  high  level  of  agreement  among  all  groups  as 

to  the  most  and  least  important  functions  of  evaluation.  Nearly 

all  groups  agree  that  the  least  important  functions  of  evaluation 

are  the  regulation  of  individual  staff  behavior  and  the  evaluation 

7 

of  individual  staff  performances.  Clinicians  and  directors  agree 
that  efficiency  and  accountability  to  the  community  are  the  next 
most  important  functions,  after  program  evaluation  and  feedback. 

In  general,  the  responses  in  Table  4. 1 support  the  view  that  the 
staff  prefers  evaluations  which  focus  on  programs  rather  than 
individuals.  Indeed,  the  evaluators  chose  the  least  threatening 
function,  feedback  for  the  clinical  staff,  as  the  most  important. 

While  the  rankings  provide  a general  overview  as  to  priorities 
that  the  different  groups  assign  to  various  functions  of  evaluation, 
more  specific  questions  were  also  asked,  particularly  with  respect 

g 

to  views  concerning  the  "worth"  of  formal  evaluation.  An  examination 
of  Tables  4.  2,  4.  3,  4.  4,  and  4.  5 shows  that  there  is  generally  strong 


^Several  of  the  functions  are  closely  associated  with  socially 
desirable  or  undesirable  terms  such  as  "accountability"  and  regulate.  " 
Again,  it  was  felt  necessary  to  include  them  because  of  their  im- 
portance despite  the  potential  response  bias.  For  example,  some 
clinicians  and  directors  argued  (during  participant  observation  and 
interviews)  that  evaluation  inevitably  involves  the  regulation  and 
control  of  the  staff. 

g 

The  questionnaire  clearly  defined  evaluation  as  formal  evaluation 
carried  out  by  an  evaluator  and  his  staff. 


122 


support  for  the  value  of  formal  evaluation.  Table  4.  2 demonstrates 
that  majorities  of  clinicians  and  directors  reject  the  argument  that 
formal  evaluation  is  likely  to  cost  more  than  it  is  worth.  Most 
clinicians  and  directors  also  support  the  idea  of  devoting  a larger 
percentage  of  the  budget  to  evaluation  as  indicated  by  Table  4.  3 
Tables  4.  4 and  4.  5 show  that  most  clinicians  and  directors  reject 
the  often  stated  arguments  that  evaluation  is  irrelevant  and  unlikely 
to  be  used  in  decision  making. 

A significant  minority,  approximately  30  percent  for  all  three 
groups,  agree  that  information  from  the  evaluations  is  not  likely 
to  be  used  in  decision  making.  The  only  other  potentially  negative 
note  concerning  evaluation  is  revealed  in  Table  4.  6.  Majorities 
of  clinicians  and  directors  agree  that  direct  services  should  take 
precedence  over  evaluation. 

In  summary,  the  overall  perspective  as  measured  by  the 
questionnaires  is  positive  toward  the  worth  of  formal  evaluation. 

There  is  especially  strong  support  for  certain  types  of  evaluation 
such  as  program  evaluation  and  evaluation  for  the  purpose  of  feedback. 
These  data  reinforces  the  point  that  the  staff  much  prefers  program 
evaluation  to  evaluation  which  focuses  on  the  individual  therapist. 

Secondly,  Table  4. 1 demonstrates  that  accountability  to  the  clientele 
of  the  organization  is  not  ranked  very  high  by  any  group  (fifth  by 
clinicians,  sixth  by  directors,  and  seventh  by  evaluators).  Accountability 
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TABLE  4.  2 

COST  OF  EVALUATION 


It  is  likely  that  evaluation  will  cost  more  than  the  information 
derived  from  it  is  worth. 


Response 

Percentage  of 
Clinicians 

Percentage  of 
Directors- 

Percentage  of 
Evaluators 

Strongly  Agree 

— 

— 

— 

Agree 

7.  9 (3) 

6.  3 (1) 

— 

Undecided 

7.  9 (3) 

18.  8 (3) 

— 

Disagree 

76.  3 (29) 

37. 5 (6) 

66. 7 (6) 

Strongly  Disagree 

7.  9 (3) 

37.  5 (6) 

33.  3 (3) 

*Sum  of  percentage  figures  is  greater  than  100  percent  because  of 
rounding. 


TABLE  4.  3 

PERCENT  OF  BUDGET  TO  EVALUATION 


A higher  percentage  of  the  total  budget  (of  the  mental  health 
organization)  should  be  devoted  to  evaluation  than  is  currently  the  case. 


Response 

Percentage  of 
Clinicians 

Percentage  of 
Directors  * 

Percentage  of 
Evaluators 

Strongly  Agree 

7.  9 (3) 

37.  5 (6) 

33. 3 (3) 

Agree 

76.  3 (29) 

37.  5 (6) 

66. 7 (6) 

Undecided 

7.  9 (3) 

18.  8 (3) 

— 

Disagree 

7.  9 (3) 

6.  3 (1) 

— 

Strongly  Disagree 

— 

— 

— 

*Sum  of  percentage  figures  is  greater  than  100  percent  because  of 
rounding 
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TABLE  4.  4 

RELEVANCE  OF  EVALUATION 


Evaluation  is  unlikely  to  produce  information  relevant  to  the 
people  performing  direct  services. 

Response 

Percentage  of 
Clinicians 

Percentage  of 
Directors* 

Percentage  of 
Evaluators* 

Strongly  Agree 

2.  6 (1) 

6.  3 (1) 

— 

Agree 

5.  3 (2) 

18.  8 (3) 

— 

Undecided 

— 

18.  8 (3) 

— 

Disagree 

73.  7 (28) 

25.  0 (4) 

55. 5 (5) 

Strongly  Disagree 

18.  4 (7) 

31.  3 (5) 

44.  4 (4) 

*Sum  of  percentage  figures  is  unequal  to  100  percent  because  of 
rounding. 


TABLE  4.  5 

USE  IN  ADMINISTRATION 


Most  of  the  information  derived  from  evaluation  is  unlikely  to 
be  used  in  actual  administrative  decision  making. 


Response 

Percentage  of 
Clinicians 

Percentage  of 
Directors  * 

Percentage  of 
Evaluators 

Strongly  Agree 

2.  6 (1) 

— 

— 

Agree 

26.  3 (10) 

31. 3 (5) 

33. 3 (3) 

Undecided 

15.  8 (6) 

25.  0 (4) 

— 

Dis  agree 

47.  4 (18) 

25.0  (4) 

66. 7 (6) 

Strongly  Disagree 

7.  9 (3) 

18.  8 (3) 

— 

*Sum  of  percentage  figures  is  greater  than  100  percent  because  of 
rounding. 
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TABLE  4.  6 

EVALUATION  AS  LOWER  PRIORITY 


Evaluation  is  necessarily  of  lower  priority  than  the 
delivery  of  services. 

direct 

Res  ponse 

Percentage  of 
Clinicians* 

Percentage  of 
Directors  * 

Percentage  of 
Evaluators* 

Strongly  Agree 

2.  6 (1) 

6.  3 (1) 

— 

Agree 

50.  0 (19) 

50.  0 (8) 

44.  4 (4) 

Undecided 

10.  5 (4) 

25. 0 (4) 

22. 2 (2) 

Disagree 

34.  2 (13) 

6.  3 (1) 

22. 2 (2) 

Strongly  Disagree 

2.  6 (1) 

12.  5 (2) 

11. 1 (1) 

*Sum  of  percentage  figures  is  unequal  to  100  percent  because  of 
rounding. 


to  community  is  ranked  as  a more  important  function  than  account- 
ability to  clientele  by  all  three  groups.  Nevertheless,  the  formal 
evaluations  presently  being  introduced  tend  to  emphasize  evaluation 
by  the  clientele  of  the  organization  such  as  goal  attainment  scaling 
and  consumer  satisfaction  surveys. 

Another  interesting  finding  is  the  fact  that  the  one  function 
that  can  be  identified  with  the  goal  of  organizational  survival,  namely 
"providing  justification  for  the  funding  of  the  mental  health  center,  " 
is  rated  low  in  importance  as  indicated  by  Table  4.  1 The  clinicians, 
directors,  and  evaluators  rank  this  "survival  function"  as  being 
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eighth,  seventh,  and  sixth  in  importance  respectively. 

Consequently,  the  above  data  suggest  that  the  functions 

regarded  as  most  important  by  members  of  these  organizations  are 

related  to  program  improvement  rather  than  personal  or  organizational 

goals  of  survival.  However,  such  conclusions  cannot  yet  be  drawn 

because  of  the  need  for  additional  information  and  the  possibility  of 

g 

bias  and  misinterpretation  in  the  responses  to  the  questionnaire.  Also, 
preferences  expressed  on  questionnaires  do  not  always  predict  or 
accurately  reflect  behaviors.  Using  data  based  on  interviews  and 
participant  observation,  the  question  of  what  role  evaluation  plays 
will  be  examined  with  respect  to  a problem  that  is  often  seen  as  being 
a serious  obstacle  to  organizational  evaluation:  the  organization's 
struggle  for  survival. 


The  potentially  strong  value  associations  of  several  of 
the  functions  have  already  been  noted.  A second,  more  serious 
problem,  is  the  possibility  that  respondents  interpreted  the  question 
(though  it  was  not  stated  as  such)  as  "What  should  be  the  functions 
of  evaluation?"  In  retrospect,  two  separate  questions  could  have 
been  asked:  (1)  What  should  be  the  function  of  evaluation?  (2) 

What  actually  is  the  function  of  evaluation?  However,  since  evaluation 
was  restricted  to  formal  evaluation  carried  out  by  an  evaluation 
staff,  the  latter  question  would  not  have  been  meaningful  to  those 
organizations  that  did  not  have  such  a staff.  Also,  it  is  hoped  that 
the  distinction  between  the  two  above  questions  is  made  by  comparing 
the  questionnaire  data  with  the  interviews  and  participant  observation 
data. 


See  Allen  P.  Wicker,  "Attitudes  versus  Action:  The 
Relationship  of  Verbal  and  Overt  Responses  to  Attitudes,  " Journal 
of  Social  Issues,  Vol.  25  (Autumn,  1969),  pp.  41-78. 
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Evaluation  and  the  Struggle  for  Survival 

The  organization's  "survival  function"  is  a broad  concept  that 
can  include  a variety  of  activities.  However,  securing  the  necessary 
funding  to  provide  for  the  salaries  and  resources  needed  by  an 
organization  is  the  prime  factor  that  one  thinks  of  as  being  necessary 
to  insure  organizational  survival. 

Organizations  differ  greatly  in  their  security  as  far  as  funding 
is  concerned,  with  some  organizations  being  wealthy  enough  to  be 
far  removed  from  any  overt  sign  of  struggle  for  survival.  There 
exists  some  variation  in  the  group  of  organizations  under  study 
concerning  their  overall  resources.  However,  there  is  strong 
evidence  that  most  of  the  fourteen  organizations  in  this  study,  and 
mental  health  organizations  in  general,  perceive  a crisis  to  be 
present  in  regard  to  obtaining  the  necessary  funding  and  insuring 
organizational  survival. 

Mental  health  centers  received  highly  favorable  budgetary 
treatment  during  most  of  the  1960's,  but  they  have  fared  less 
favorably  in  recent  years. 11  Pre-planned  decreasing  federal 
contributions  have  forced  these  organizations  to  seek  funds  from 
sources  other  than  the  federal  government.  Most  states  are  having 

11See  Constance  Holden,  "Mental  Health:  NIMH  Reeling  Over 
Proposed  Budget  Cuts,  " Science,  April  20,  1973,  pp.  284-285. 
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their  own  funding  difficulties  and  the  clientele  of  most  of  the  centers 
is  relatively  poor.  As  a consequence,  there  is  anxiety  concerning 
funding.  This  anxiety  was  reflected  repeatedly  in  the  interviews 
and  participant  observation.  In  the  following  illustration,  the  business 
manager  of  a mental  health  organization  reveals  that  salaries  for 
the  forthcoming  year  were  not  even  covered  by  funds  which  the 
organization  could  depend  on: 

(Business  Manager)  The  salaries  for  next  year  will 
cost  $904,000.00  while  income  for  the  salaries  is  only 
$78,  000.  00  plus  a little  more  for  secretarial  personnel. 

One  solution  is  not  to  fill  positions  as  quickly  as  people  leave. 

(Director)  We  are  not  an  expanding  operation  as  we 
look  to  the  future. 

(Supervisory  Clinician)  Then  we  should  not  think  in 
terms  of  growth  and  development  but  of  contracting  and 
atrophying. 

During  an  interview  with  a clinician  responsible  for  the  programs 
in  a certain  area,  a member  of  the  county  commission  came  in  to 
deliver  what  the  commissioner  himself  called  a "bombshell.  " 

He  said  that  the  county  commission  had  voted  not  to  go  along  with 
what  had  been  a pre-planned  increase  in  the  county's  contribution 
to  the  funding  of  the  mental  health  organization.  Among  the  reasons 
why  the  commission  had  refused  to  go  along,  the  commissioner 
cited  resentment  of  programs  "controlled  by  Washington"  and  a 
feeling  that  mental  health  funds  were  the  most  easily  cut: 
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(County  Commissioner)  There  was  a conscious  decision 
not  to  raise  the  millage  rate,  that  the  money  had  to  be  cut 
from  the  budget  as  a result,  and  that  there  should  be  a conscious 
decision  to  use  revenue  sharing  dollars  for.  . . activities  like 
landfill  operations  and  building  roads.  They  thought  that  the 
Mental  Health  Center  was  the  prime  thing  that  should  be  cut. 

I argued  against  the  cut  but  some  opposed  it  because  they  are 
opposed  to  anything  out  of  Washington,  you  know,  and  others 
opposed  it  on  the  basis  that  it  was  the  easiest  thing  to  cut. 

Further  evidence  of  the  "hard  times"  atmosphere  present 

in  these  organizations  occurred  in  a discussion  at  a Center  where 

clinicians  had  been  complaining  about  the  vast  amount  of  paperwork 

required  by  the  State  in  order  to  obtain  funds.  "Outside  pressures" 

were  referred  to  in  the  argument  used  by  the  evaluator  in  reply 

to  the  complaints: 

(Supervisory  Clinician  #1)  You  are  not  going  to  get 
less  paperwork  through  a contractual  set-up.  ^ The  whole 
process  of  evaluation  depends  on  paperwork.  But  I share  his 
[member  of  the  evaluation  staff]  feeling  about  outside  pressures 
and  the  need  for  records.  I was  in  the  position  of  justifying  a 
$250,  000.  00  program  suddenly.  Even  though  I don't  like  doing 
them,  they  are  necessary  and  the  question  is  how  to  make  them 
easier  and  more  relevant. 

(Evaluator)  The  State  wants  answers  to  each  of  the 
questions  on  this  intake  form. 

(Supervisory  Clinician  #2)  I am  not  aware  of  what  the 
outside  pressures  are.  I am  afraid  to  ask  but  I would  like  to 
know. 

i 

(Director)  They  concern  the  increased  tendency  to 
require  payment  for  specific  things,  not  just  for  an  hour  of 


i o 

They  had  been  discussing  the  possibility  of  each  program 
establishing  a contract  with  the  organization  to  achieve  a more 
decentralized  yet  "accountable"  form  of  administration. 
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therapy  but  for  a consequence  or  outcome.  You  will  have  to 
have  an  evaluation  of  exactly  what  therapy  achieves. 

(Supervisory  Clinician  #3)  They  have  been  trying  to  do 
that  for  100  years,  like  in  the  highway  department.  . . . 

One  final  example  of  the  concern  with  survival  will  suffice. 

In  this  case,  a director  of  a program  describes  how  he  had  made 

out  several  budgets  ranging  from  optimistic  to  pessimistic.  His 

comments  arouse  a discussion  concerning  the  future  of  funding  for 

mental  health  organizations: 

(Supervisory  Clinician  #1)  I have  five  maps  [budgets] 
and  if  they  don't  like  one,  I throw  it  out  and.  . . . 

(Supervisory  Clinician  #2)  But  this  is  the  big  question: 

Is  there  a future  in  mental  health,  let's  say  two  years  from  now? 

(Evaluator)  We  are  su  osed  to  be  so  successful  that 


we  don't  need  support.  . . . 

(Supervisory  Clinician  #3)  But  that  is  just  an  excuse.  . . . 

(Director)  But  the  proposed  national  health  insurance 
schemes  may  provide  funds.  . . . 

To  summarize,  the  organizational  struggle  for  survival  is 
no  abstract  phrase  to  the  personnel  of  these  organizations.  Concern 
with  survival  is  evident  in  their  anxiety  about  the  budgets  of  programs, 
about  their  salaries,  and  even  about  the  future  security  of  their  jobs. 

1 3 

This  phrase  was  used  several  times  by  people  in  the  field 
and  apparently  originated  in  the  Nixon  administration.  But  it  was 
used  in  an  ironic  sense  to  imply  that  the  Nixon  administration  was 
really  trying  to  sabotage  them  through  this  kind  of  praise. 
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Moreover,  the  struggle  for  survival  had  the  effect  of  creating 
tension  and  minor  struggles  among  programs  and  even  individual 
staff  members  for  funds  and  salaries.  Due  to  a lack  of  funds, 
struggles  occur  between  different  programs  when  heads  of  programs 
find  that  they  cannot  expand  their  service.  Expansion  for  one  program 
can  only  be  achieved  by  a cut  in  the  funding  of  another  program  as 
the  following  case  illustrates: 

(Program  Head)  What  we  [the  staff  of  the  program  he 
headed]  are  asking  for  is  a reallocation  of  staff. 

(Director)  You  have  to  be  conscious  of  the  real  world. 
Whose  staff  are  you  going  to  take  away? 

(Program Head)  But  we  still  have  to  look  at  the  staffing 
patterns  for  different  programs. 

(Director)  I think  that  most  of  the  areas  are  going  to 
say  that  they  don't  have  enough  staff  right  now. 

(Program  Head)  Well,  I will  have  to  go  back  and  redo 
my  budget  again. 

(Director)  One  of  the  assumptions  that  I have  made 
is  that  we  don't  fire  lightly  or  move  staff  from  one  program  to 
another,  because  of  the  problem  of  morale. 

(Program  Head)  You  mean  to  tell  me  that  we  are  not 
going  to  look  at  the  staffing  pattern  when  it  was  arbitrarily  set? 

Stringency  of  resources  brings  out  divisions  among  both 

personnel  and  programs  in  organizations.  In  such  situations, 

programs  attempt  to  prove  their  worth  and  excellence.  In  one  case, 

a program  director  used  the  annual  report  of  the  organization  to  make 

detailed  comparisons  between  the  program  he  headed  and  similar 


132 


programs  throughout  the  county.  With  data  such  as  staff -to -patient 
ratios,  his  report  cited  figures  which  showed  that  his  program  was 
carrying  a much  greater  burden  than  comparable  programs  in 
other  mental  health  organizations.  The  same  report  also  introduced 
several  other  kinds  of  information  to  show  that  this  particular 
program  had  performed  well,  with  the  implication  being  that  it 
should  be  rewarded  by  increased  funding.  In  short,  the  program 
head  had  created  an  ad  hoc  evaluation  of  his  own  program  in  order 
to  show  why  it  should  receive  more  funds  and  staff  despite  the 
scarcity  of  these  resources. 

The  above  cases  illustrate  the  fact  that  the  struggle  for  survival 
in  these  organizations  often  leads  to  increased  tension  in  organizations. 
Concern  with  survival  creates  a situation  in  which  programs  are 
under  pressure  to  justify  the  amount  of  money  spent  on  them.  Under 
these  conditions  of  anxiety  and  tension,  it  would  be  impressive 
support  for  the  concept  of  the  "self-evaluating"  organization  if 
evaluations  were  carried  out  to  fulfill  idealistic  functions  to  which 
the  staff  members  had  assigned  the  highest  priority,  such  as  program 
evaluation,  feedback,  accountability  to  community,  and  efficiency. 

The  Impetus  for  Formal  Evaluation 

Directors,  clinicians,  and  evaluators  were  specifically  questioned 
as  to  what  the  motivations  behind  the  movement  for  evaluation  are. 
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It  is  interesting  to  examine  first  the  responses  of  the  directors 
and  personnel  of  those  centers  which  had  decided  not  to  invest  in 
a formal  evaluation  staff,  even  though  many  had  considered  doing 
so.  The  basic  reasoning  behind  their  decision  was  that  an  evaluation 
staff  would  cost  money  which  they  could  not  afford.  One  director 
openly  states  that  an  evaluator  would  not  be  "cost  effective": 

(Director)  This  cost  is  difficult  to  justify.  We  originally 
had  a full-time  research  person  written  into  our  grant.  We 
costed  it  out,  and  the  amount  of  service  did  not  justify  the 
dollars  cost.  I didn't  see  how  we  could  do  it.  We  went  the 
route  of  being  primarily  service-oriented,  trusting  that  somehow 
we  were  doing  some  good. 

Another  director  notes  that  the  decrease  in  funds  received  from  the 
federal  government  changed  his  mind  about  hiring  an  evaluator: 

(Director)  I think  that  we  need  somebody  in  the  organization 
who  has  education,  background,  or  experience  in  research  and 
evaluation  but  now  we  have  gotten  to  the  point  in  our  grant  that 
we  see  clinical  needs  are  more  important --this  was  our  last 
discussion  concerning  the  matter  [ evaluation] --that  we  may  not 
pursue  this  any  further.  We  have  so  many  clinical  needs.  Why 
should  we  flounder  around,  especially  since  nobody  else  seems 
to  be  doing  anything  much?  So  why  should  we  go  head  over  heels? 

Not  only  do  many  staff  members  in  these  organizations  feel 

that  clinical  service  is  the  primary  mission  of  the  organization 

but  also  that  clinical  services  provide  the  organization  with  money 

in  the  form  of  patient  fees  or  through  reimbursement  from  the 

State  for  certain  categories  of  clients  unable  to  pay  their  fees. 

A clinician  in  one  organization  which  was  not  doing  any  formal 

evaluation  describes  the  conditions  which  might  lead  the  organization 


to  establish  an  evaluation  program: 
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(Supervisory  Clinician)  We  have  the  capacity  and 
capability  of  shifting  from  a service-oriented  facility  to  a 
training  or  even  research  facility,  of  course,  if  ordered  by 
whoever  orders  that  sort  of  thing.  Several  of  us  could  consider 
that  if  it  were  necessary  for  us  to  do  it  in  order  to  eat.  Then 
we  could  certainly  do  it. 

Even  in  organizations  with  evaluation  units,  it  is  obvious  that 
the  costs  of  evaluation  had  been  carefully  considered,  as  the  following 
director  indicates  in  his  response  to  a question:  "Can  you  evaluate 
evaluation? 

(Director)  Well,  of  course.  For  instance,  one  time 
our  district  board  was  interested  in  asking  the  State  for  a 
quarter  of  a million  dollars  for  a research  project  when  our 
total  budget  was  only  a million  dollars.  I questioned  spending 
a quarter  million  dollars  to  evaluate  a million  dollars'  worth 
of  service.  . . but  one  is  concerned  with  the  cost  of  evaluation 
and  research.  However,  if  it  [evaluation]  earns  money,  in 
other  words,  if  it  participates  in  putting  money  into  the  Center 
from  the  community  or  through  research  and  evaluation  grants, 
at  least  from  a practical,  capitalistic  point  of  view,  it  is 
paying  for  itself. 

Thus,  pragmatic  concerns  with  costs  of  evaluation  led  several 
organizations  to  decide  against  investing  in  an  evaluation  staff. 

What  are  the  positive  inducements  which  led  certain  organiza- 
tions to  invest  in  formal  evaluation  and  an  evaluation  staff?  What 
might  make  directors  think  that  evaluators  are  cost  effective? 

Interviews  and  participant  observation  show  that  the  basic  impetus 
behind  the  movement  toward  evaluation  is  related  to  outside  forces 
important  to  the  organization's  survival.  Two  points  were  continually 
emphasized  in  interviews  and  participant  observation:  (1)  Organizational 
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members  expect  the  problems  of  funding  to  continue;  and  (2)  they 
perceive  a growing  movement  toward  accountability  and  the  need 
to  justify  expenditures  on  mental  health.  Typical  comments 
concerning  the  impetus  in  the  movement  toward  formal  evaluation 
include  the  following: 

(Evaluator)  Well,  of  course,  I think  that  there  is  pressure 
from  the  government.  You  know  they  want  to  know  that  the 
money  they  are  spending  is  doing  some  good.  I think  that  in 
general  any  sort  of  human  service  agency  is  becoming  more 
accountable  to  its  clientele  now,  because  people  are  saying, 
"Alright,  damn  it,  our  taxes  are  paying  for  this  and  that  and 
everything  you  want  from  us.  So  let's  see  some  evidence  that 
what  you  do  for  these  people  is  making  a difference.  " 

(Director)  Really,  the  main  impetus  [for  evaluation] 
had  been  outside  forces.  The  board  of  the  Center,  for  one, 
wants  to  know  what  we  are  doing,  and  NIMH  is  asking  us 
to  be  more  accountable.  Anyone  who  is  receiving  govern- 
ment funds  knows  that  there  is  obviously  a big  push  on  for 
accountability  and  a little  more  than  that,  not  just  to  account 
for  hours,  but  whether  or  not  you  are  really  doing  something. 
This  is  the  impetus  as  I see  it.  . . . 

(NIMH  Official)  We  don't  know  what  the  future  holds.  . . 
With  revenue  sharing,  the  county  commissioners  and  the  cities 
are  going  to  get  at  some  dollars.  This  calls  for  an  additional 
emphasis  to  do  something  in  evaluation  efforts --to  sell  them 
to  the  community  to  get  their  support.  . . . There  will  be  those 
looking  to  see  if  they  can  get  more  bang  for  the  buck. 

(State  Official)  Evaluation  and  cost  effectiveness  are 
what  the  mental  health  boards  are  talking  about.  . . . These 
boards  want  to  talk  about  evaluation  in  a different  sense  from 
what  evaluators  are  usually  thinking  of,  that  is  how  do  we 
justify  the  money  being  used  to  the  community? 

Even  those  centers  which  aren't  doing  very  much  as  far  as 

formal  evaluation  is  concerned  recognized  these  outside  pressures. 

In  one  such  organization,  the  author  asked  the  following  clinician 
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if  all  of  the  talk  and  interest  in  formal  evaluation  is  just  a fad: 

(Clinician)  No,  it  is  economics.  . . I think  the  govern- 
ment, politicians,  and  everybody  are  concerned  about  that 
money.  They  want  to  know  if  I am  spending  money,  "How  do 
I justify  it?  What  am  I getting  for  my  dollar?"  Which  are 
perfectly  valid  questions.  . . . 

A director  who  is  highly  skeptical  of  evaluation  predicted  that  the 
demise  of  formal  evaluation  would  come  if  and  when  government 
funding  ceases: 

(Director)  I can  tell  you  exactly  when  it  [evaluation] 
will  evaporate --when  government  money  runs  out.  That  is  the 
reason  we  do  it,  because  we  are  forced  to  do  it  in  order  to  get 
government  funds.  We  are  in  a continuous  process  of  what  I 
call  evaluation.  They  beg  to  differ  in  many  instances  but  all 
of  these  forms--paper  that  they  send  out--are  forms  of  evalua- 
tion, and  they  ask  for  statistics.  You  fill  them  out  and  that  is 
just  as  much  an  evaluation  for  me  as  a balance  sheet  or  income 
statement.  ...  I can  promise  you  that  when  this  continuation 
grant  application  runs  uut  we  will  no  longer  continue  to  write 
a continuation  grant  type  of  evaluation.  It  is  too  time  consuming. 
It  costs  a fortune  in  man-hours  and  you  don't  even  use  it. 

Thus,  there  is  a great  deal  of  agreement  that  outside  forces 

and  concern  for  organizational  survival  provide  the  underlying 

motivation  for  the  movement  to  carry  out  formal  evaluation  of 

mental  health  organizations,  programs,  and  personnel.  Indeed, 

the  frankness  which  characterized  most  responses  to  questions 

concerning  this  point  merely  reflects  the  fact  that  center  personnel 

consider  these  facts  to  be  common  knowledge,  at  least  in  the  mental 

health  field. 


The  Benefits  of  Evaluation 


Although  there  have  been  some  indications  of  the  benefits  of 
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evaluation  in  the  comments  above,  it  still  remains  to  be  shown 
in  detail  exactly  how  formal  evaluation  contributes  to  organizational 
survival.  At  least  four  categories  of  actual  or  expected  benefits 
from  formal  evaluation  were  identified:  (1)  evaluation  as  a defensive 
strategy  in  anticipation  of  future  requirements  for  evaluation  and 
accountability  by  funding  agencies  and  district  boards,  (2)  evalua- 
tion as  a means  of  providing  justification  of  the  worth  of  mental  health 
services  to  the  community,  (3)  evaluation  as  a method  of  securing 
grants  and  other  resources  for  the  organization  and  individuals 
within  it,  and  (4)  the  use  of  evaluation  staff  to  carry  out  "non- 
evaluative"  activities  necessary  or  useful  to  the  organization's 
survival. 

Evaluation  and  anticipation 

Evidence  was  cited  in  the  introductory  chapter  that  mental 
health  organizations  appear  to  be  devoting  a much  greater  amount  of 
attention  and  resources  than  most  other  public  organizations  to 
formal  evaluation.  One  of  the  reasons  for  this  fact  appears  to  be 
that  members  of  organizations  in  this  field  are  carefully  scanning 
the  environment  to  find  methods  of  helping  their  organizations  to 
survive.  They  see  a chance  to  get  a headstart  on  what  they  perceive 
to  be  an  inevitable  movement  toward  a requirement  for  formal 
evaluation  of  all  service  programs.  Thus,  when  the  author  asked 
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one  evaluator  why  the  director  of  the  Center  had  hired  him,  the 
evaluator  replied: 

(Evaluator)  Well,  as  he  [the  director]  expressed  it 
to  me,  it  was  a case  of  doing  something  that  was  going  to 
come  anyway  and  that  he  would  rather  have  our  Center  in  the 
forefront  of  the  movement. 

Another  evaluator  expresses  a similar  thought,  although  in  a 
more  cynical  tone,  in  response  to  the  same  question: 

(Evaluator)  That  is  a good  question.  At  the  risk 
of  being  unfair,  he  saw  its  major  function  as  demonstrating  to 
Tallahassee  that  we  were  a cued-in  bunch  down  here,  that  we 
were  on  top  of  it,  and  out  front  with  the  best  and  most  progressive 
kinds  of  techniques,  which  meant  that  we  should  get  our  fair 
share  of  the  money. 

Voluntarily  undertaking  evaluation  is  seen  as  a means  of 
achieving  greater  input  and  control  over  the  forms  of  evaluation 
likely  to  be  imposed  later: 

(Director)  The  community  is  doing  it  [evaluation]. 

It  is  saying,  "Do  I want  my  United  Fund  dollar  to  go  here?  " 
or  "Do  I want  my  tax  dollar  going  here?  " So  whether  you  think 
it  is  being  done  fairly  or  not,  it  is  being  done.  And  if  somebody 
else  is  going  to  do  it,  you  might  as  well  do  it  yourself,  coming 
up  with  some  kinds  of  guidelines  to  feed  back  to  them.  If 
we  don't  do  it,  somebody  else  will  do  it  for  us.  . . . 

A final  example  of  this  use  of  evaluation  occurs  in  the  following 
exchange  between  a program  head  and  a director.  They  are  dis- 
cussing the  reasons  for  attempting  to  put  together  an  extensive 
program  budget.  In  this  budget,  different  programs  would  promise 
to  provide  a certain  number  of  service  units  per  dollar: 

(Program  Head)  Is  this  a fee-for-service  thing?  Are 
we  going  to  use  this  as  a justification  for  getting  money  and 
not  just  for  the  board? 
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(Director  We  are  trying  to  get  a jump  on  -what  will  be 
required  of  us  in  the  future.  . . . You  can  say  you  are  going 
to  get  paid  for  each  patient  cured.  . . . 

(Program  Head)  If  that  is  the  case,  we  are  in  trouble 
in  this  business.  . . . 

The  question  arises  concerning  the  reason  why  mental  health 
organizations  are  unusually  concerned  with  getting  a headstart 
on  formal  evaluation.  They  are  not  the  only  group  of  organizations 
which  have  experienced  funding  difficulties.  Their  concern  with 
getting  a headstart  on  future  requirements  seems  to  be  the  result 
of  the  interaction  between  two  factors:  (1)  the  fact  that  they  perceive 
themselves  to  be  in  funding  difficulties,  (2)  the  fact  that  they  are 
composed  substantially  of  highly  educated  professionals  who  care- 
fully scan  the  environment  in  order  to  anticipate  trends  that  might 
affect  or  benefit  their  organizations.  Organizations  with  fewer 
problems  in  funding  do  not  have  to  be  so  concerned  with  the  future. 
Ones  composed  of  leadership  with  less  experience  and  capability  of 
scanning  the  environment  and  anticipating  the  future  are  not  likely 
to  give  evidence  of  so  much  concern  for  evaluation  until  it  is 
actually  required  of  them. 


14Another  example  of  an  organization  with  a similar  combina- 
tion of  conditions  is  NASA  which  has  made  a strong  attempt  to 
prove  the  worth  of  its  technology  to  areas  other  than  the  space 
program,  particularly  after  it  began  to  fare  less  favorably  in 
appropriations.  See,  e.  g.  , Raymond  A.  Bauer,  Second  Order 
Consequences:  A Methodological  Essay  on  the  Impact  of  Technology 
(Cambridge:  M.  I.  T.  Press,  1969). 
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It  is  also  interesting  to  draw  an  analogy  between  the  actions 
taken  by  mental  health  organizations  as  a whole  and  the  behavior 
of  the  evaluators  within  them.  As  noted  earlier,  most  of  the 
evaluators  had  anticipated  difficulty  in  their  role  (perhaps  because 
of  reading  or  hearing  about  the  problems  of  doing  research  and 
evaluation  in  service -oriented  organizations).  They  undertook 
non -threatening  evaluations  in  order  to  prevent  any  difficulty.  It 
seems  that  the  rule  of  anticipated  reactions  is  particularly  important 
in  situations  when  educated  and  intelligent  individuals  or  groups  of 
individuals  face  difficult  and  uncertain  situations. 

Evaluation  and  the  worth  of  mental  health  services 

While  several  directors  and  clinicians  are  pleased  to  undertake 
evaluation  in  order  to  get  a headstart  on  other  organizations,  they 
also  expect  the  evaluators  and  formal  evaluation  to  contribute  some- 
thing of  more  immediate  value  to  the  organization.  In  particular, 
they  see  evaluation  as  a means  of  justifying  the  money  already 
spent  on  mental  health  and,  if  possible,  providing  justification  for 
asking  for  even  more  funds. 

In  discussing  the  popularity  of  psychiatric  epidemiological 
surveys,  it  was  noted  that  these  surveys  are  often  expected  to 
show  a great  degree  of  prevalence  of  mental  illness.  Thus,  they 
are  likely  to  furnish  "hard  data"  to  support  requests  for  more 
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mental  health  service.  A good  illustration  of  this  point  occurred 
at  a meeting  attended  by  the  author.  A researcher  presented  a 
detailed  critique  of  psychiatric  epidemiological  surveys.  He 
argued  that  they  are  virtually  worthless  for  making  decisions 
concerning  the  provision  of  mental  health  services.  In  reply,  one 
listener,  who  is  a member  of  a mental  health  district  board,  replied 
briefly:  "You  have  to  have  incidence  to  get  dollars.  " 

An  evaluator  describes  how  she  thought  that  the  director  of  the 
organization  conceptualizes  her  role: 

(Evaluator)  I think  that  [name  of  the  director]  wanted 
information  to  back  up  his  arguments  on  money  that  he  had  to 
make  to  people  concerning  the  Baker  Act.  For  example, 
does  the  Baker  Act  really  mean  that  we  are  taking  the  burden 
off  the  local  police  department,  the  local  hospitals,  and  pro- 
viding more  local  care?  And  he  did  not  have  the  mechanism 
to  get  this  information  before  I was  here.  . . . 

A director  describes  how  he  is  using  his  evaluation  staff  to  see  if 

the  Center  is  getting  its  fair  share  of  funds  from  the  State: 

(Director)  I don't  know  if  we  can  do  this,  but  we  have 
gotten  some  statistics  together  which  we  think  that  we  can 
use  for  the  State  when  we  question  their  funding.  . . since  the 
adjoining  county  received  $800,  000.  00  for  the  Baker  Act  and 
we  only  got  $1 50,  000.  00  to  $200,  000.  00  in  our  county  when  we 
had  an  extremely  high  rate  of  admissions.  . . . 

A supervisory  clinician  responsible  for  administering  mental 

health  services  in  a certain  area  got  together  statistics  demonstrating 


■*‘5The  "Baker  Act"  is  the  title  given  to  legislation  related  to 
the  funding  of  mental  health  centers  in  the  State  of  Florida. 
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the  worth  of  mental  health  services  to  the  county.  The  evaluator 
in  the  organization  indicated  the  he  was  interested  in  seeing  how  the 
- clinician  had  arrived  at  these  figures.  " The  figures  cited  by  the 
clinician  included  the  following  points: 

It  is  estimated  that  $10,000  worth  of  services  has  been 
provided  to  the  sheriff's  department  and  county.  Specific 
services  can  be  verified  including  the  following:  (1)  a minimum 
of  forty  days  jail  time  saved  on  mental  patients.  . . (2) 
evaluation  and  counselling  ordered  through  the  Division  of 
Youth  Services  is  calculated  at  $300.  00.  . . . $10,  000  worth  of 
service  has  been  provided  to  the  county  schools.  In-service 
training  and  consultation  have  been  provided  in  seven  of  the 
county  schools.  . . . 

Many  of  these  ad  hoc  evaluations  are  collections  of  statistics 
put  together  to  show  the  services  of  the  organizations  in  as  favorable 
a light  as  possible.  No  cases  could  be  found  of  mental  health 
organizations  deliberately  presenting  information  that  reflected 
unfavorably  on  their  work.  Yet,  some  potentially  negative  statistics 
were  available  for  some  of  the  organizations.  For  example,  the 
mental  hospital  load  and  inpatient  figures  had  failed  to  decrease  in 
many  areas  despite  the  new  mental  health  act  which  established 
support  for  community  mental  health  centers.  iD 


*®Of  course,  as  noted  in  Chapter  III,  all  evaluations  and  sta- 
tistics have  some  ambiguity  associated  with  them.  The  failure  of 
these  figures  to  decrease  was  seen  by  many  in  the  field  as  merely  the 
consequence  of  the  fact  that  the  mental  health  organizations  were 
simply  discovering  cases  which  had  previously  gone  undetected. 
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The  pressure  to  generate  data  that  show  that  the  costs  of 
mental  health  service  are  justified  is  present  even  in  those  organi- 
zations that  have  no  formal  evaluation  staff,  as  the  following  case 
illustrates.  The  director  of  this  Center  speaks  enviously  of 
rehabilitation  programs  and  their  public  relations: 

(Director)  The  model  of  successful  public  relations  is 
rehab  [rehabilitation  programs].  They  worked  out  a ratio 
of  the  amount  of  taxes  paid  as  a result  of  rehab  and  then  the 
amount  of  the  costs  of  the  programs,  and.  . . the  amount 
of  taxes  paid  by  the  people  [the  people  rehabilitated  by 
the  program]  was  twelve  times  the  amount  of  the  cost.  So 
rehabilitation  has  anexcellent  relationship  with  Congress 
while  mental  health  has  never  generated  such  positive  find- 
ings. We  have  only  generated  a sad  story.  . . . That  is  one 
of  the  reasons  why  I am  interested  in  getting  data  on  employ- 
ment of  people  served  by  our  agency.  We  have  no  idea  of  how 
many  people  have  gotten  jobs. 

While  most  staff  members,  including  clinicians  and  evaluators, 
accept  the  need  to  generate  such  data,  there  is  clearly  no  great 
enthusiasm  for  collecting  them.  The  data  lack  rigor  and  relevance, 
being  regarded  as  highly  questionable  at  best.  One  clinician  who  had 
participated  in  collecting  such  data  describes  his  attitudes  toward  it: 

(Supervisory  Clinician)  Of  course,  that  is  invalid.  I 
was  involved  in  a situation  where  we  tried  to  show  for  x amount 
of  dollars  spent  how  much  we  had  saved.  How  are  you  going 
to  do  that?  We  got  on  the  radio  and  then  after  we  left  the  studio 
of  the  place,  we  tore  up  the  notes  so  that  the  poor  other  guy 
coming  along  later  wouldn't  have  to  defend  the  figures. 

Clearly,  these  kinds  of  activities  are  not  open  evaluations 

aimed  at  measuring  the  degree  of  goal  attainment.  They  are 

calculated  to  demonstrate  positive  results.  The  willingness  to 
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collect  this  kind  of  data  rests  on  assumptions  such  as  the  following: 
(1)  that  the  costs  of  the  organization  are  justified,  even  if  it  is 
difficult,  if  not  impossible,  to  demonstrate  this  worth  in  dollars 
and  cents  figures,  and  (2)  that  it  is  necessary  to  present  the  organi- 
zation in  as  good  a light  as  possible. 

Evaluation  and  grants 

In  addition  to  giving  organizations  a headstart  and  helping  to 

justify  the  dollars  spent  on  mental  health,  evaluation  contributes 

to  the  survival  of  the  organization  by  bringing  in  grants.  Many  of 

the  evaluation  programs  are  either  receiving  or  expect  to  receive 

direct  support  from  the  state  and  federal  governments.  One  of  the 

evaluators  admitted  the  importance  of  this  money  when  asked  if  the 

amount  of  money  a Center  invested  in  evaluation,  could  be  used  as 

17 

a good  index  of  the  organization's  interest  in  self-evaluation: 

(Evaluator)  Not  necessarily.  If  somebody  could  give 
us  a grant,  and  the  Center  may  not  have  any  particular 
interest  or  investment  in  evaluation,  but  if  somebody  gave  us 
the  grant  to  do  it,  then  I think.  . . it  would  create  interest 
in  the  field.  . . . 

A state  official  attempting  to  implement  an  evaluation  program 
indicated  that  the  promise  of  a share  in  the  money  from  a grant  had 


1 7 

This  example  illustrates  the  difficulty  of  finding  rigorous 
data  to  test  Wildavsky's  question.  Certainly  the  amount  of  money 
spent  on  evaluation  is  not  a good  index  of  desire  to  carry  out 
critical  self-examination  when  the  money  is  derived  from  external 
sources. 
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persuaded  a director  of  a Center  initially  opposed  to  evaluation  to 
go  along  with  the  program.  A director  of  another  organization, 

. after  hearing  several  speeches  on  evaluation  by  directors  of  organi- 
zations undertaking  formal  evaluation,  questioned  their  motives. 

He  said  that  he  knew  their  "hearts  were  pure"  but  there  was  the 
probability  that  desire  for  funds  had  stimulated  much  of  their  interest 
in  carrying  out  formal  evaluation. 

In  addition  to  grants  which  are  specifically  tied  to  evaluation 
activities,  many  of  the  grants  received  from  NIMH  or  other  sources 
now  require  that  proposals  have  an  evaluation  section  included  within 
them.  For  example,  if  an  organization  wishes  to  get  a grant  for 
services  related  to  treatment  of  alcoholism,  it  probably  has  to 
include  within  the  grant  a method  for  evaluating  the  effects  of  the 
program.  To  summarize,  much  of  the  self-evaluation  being  under- 
taken by  the  mental  health  organizations  under  study  is  either 
directly  supported  by  outside  funds  or  is  being  undertaken  with  the 
expectation  of  receiving  such  support  in  the  future.  Thus,  it  is 
impossible  to  determine  how  much  interest  they  would  have  expressed 
in  formal  evaluation  if  they  did  not  expect  to  receive  outside  support.  ^ 

^Some  people  in  the  evaluation  field  argue  that  inexpensive 
evaluations  that  are  fairly  rigorous  and  comprehensive  can  be  carried 
out.  See  Allan  Beigel,  Evaluation  on  a Shoestring;  A Suggested 
Methodology  for  the  Evaluation  of  Community  Mental  Health  Services 
Without  Budgetary  and  Staffing  Support,  Program  Evaluation  Series, 
No.  5 (San  Francisco:  University  of  California,  November,  19  73). 
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Changes  and  innovations  in  organizations  often  result  from  "extra" 
or  slack  resources,  whether  these  are  obtained  from  the  outside 
or  are  derived  from  the  strong  financial  status  of  particular  organi- 
zations. 1 ^ 


Evaluators  and  non -evaluative  duties 


The  fourth  major  category  of  benefits  from  undertaking  formal 

evaluation  accrues  only  to  those  organizations  that  have  formal 

evaluation  staffs.  These  staffs  perform  a large  number  of  non- 

evaluative  tasks  in  the  organization  such  as  data  gathering,  general 

20 

administration,  planning,  and  other  duties.  The  following  evaluator 
describes  some  of  the  common  activities  that  he  or  his  staff  under- 
take in  addition  to  their  evaluative  tasks: 

(Evaluator)  I do  a lot  of  service  kinds  of  things  for 
people  here,  because  it  seems  to  work  really  very  well  if 
you  do.  For  example,  the  administrative  assistant  will  come 
in  and  say,  "Gee,  we  have  a board  meeting  and  wouldn't  it 
be  great  if  you  could  get  such  and  such  information?"  Whoever 
[on  the  evaluation  staff]  has  the  time,  will  work  on  that  too, 
because  that  would  be  something  that  they  need  tomorrow  or 
today  or  an  hour  from  now.  And  these  kinds  of  things  pop  up 
all  of  the  time. 


•^See  Victor  A.  Thompson,  Bureaucracy  and  Innovation 
(University:  University  o f Alabama  Press,  1968),  pp.  40-45. 

^OSome  of  these  tasks  include  gathering  data  that  are  used  in 
evaluative  ways  by  outside  agencies  but  they  do  not  conform  to 
our  definition  of  formal  evaluation  and  are  not  used  for  internal 
evaluation  purposes. 
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The  evaluator  went  on  to  point  out  that  most  of  the  material  the 
evaluation  staff  collected  is  "really  simple-minded"  and  descriptive 
in  nature  (e.  g.  , the  number  of  persons  who  attended  the  clinic 
during  a certain  period). 

A member  of  the  evaluation  team  at  another  Center  describes 
the  kinds  of  activities  which  she  had  been  doing  for  most  of  her 
career  in  this  organization: 

(Evaluator)  I used  to  keep  track  of  all  of  the  admissions, 
discharges,  types  of  therapy  and  all  of  that  which  is  now  on  the 
computer.  And  I just  found  myself  fed  up  to  here.  I was  at 
wits  end  because  they  were  trying  to  do  this  or  that  kind  of 
report  and  were  calling  me  up.  . . . I would  have  to  go  through 
the  papers  for  every  day  of  the  year.  ...  I just  didn't  see 
that  as  very  important. 

Many  of  the  clinicians  in  organizations  with  formal  evaluation  staffs 
also  see  the  evaluators  as  being  generally  involved  in  non -evaluative 
tasks: 


(Supervisory  Clinician  #1)  I have  seen  the  [department 
responsible  for  evaluation]  as  being  involved  almost  entirely 
in  administration  functions,  and  it  has  not  been  really  able 
to  begin  any  research.  . . . 

(Supervisory  Clinician  #2)  I would  like  to  see  them  [the 
evaluation  staff]  give  us  programmatic  suggestions.  In  the  first 
year,  they  seemed  to  be  caught  up  in  gathering  data  and  re- 
porting to  the  board,  such  as  with  the  alcoholism  thing.  . . . 

A clinician  in  another  organization  sees  the  role  of  the  evaluators 

as  providing  descriptive  data  which  the  clinicians  and  program  heads 


interpret: 
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(Supervisory  Clinigian)  They  [the  evaluators]  did  a 
pretty  good  job  with  keeping  up  with  the  programs  and  telling 
us  how  many  patients  had  come  in.  For  instance,  in  one  company, 
we  weren't  receiving  as  many  patients  as  we  thought  we  should 
and  we  went  back  and  asked  them  why.  ...  If  the  researcher 
hadn't  come  back  and  said  that  we  were  only  getting  forms  from 
the  wives  or  self-referrals  rather  than  referrals  from  the 
management  of  the  company,  the  therapists  might  not  keep  up 
with  this  information.  . . . We  went  back  to  the  industry  and 
found  a block  there--a  personnel  manager  who  just  wasn't 
enforcing  the  referral  system. 

Thus,  much  of  the  work  done  by  evaluators  is  either  involved  with 
the  details  of  administration  or  providing  descriptive  data  to 
clinicians. 

There  is  a scarcity  of  people  with  the  time  and  skills  to  carry 
out  the  many  data  gathering  and  administrative  duties  in  these 
organizations.  The  role  of  the  research  and  evaluation  staffs  is 
often  seen  as  being  to  help  fill  this  void: 

(Evaluator  #1)  Well,  [name  of  the  director]  has  his 
own  agenda,  and  in  his  agenda  anything  that  I can  do  to  take 
weight  off  of  his  shoulders  in  terms  of  administrative  duties, 
especially  in  his  relationships  with  the  State  and  board  and  the 
like,  is  of  primary  value  to  him. 

(Evaluator  #2)  I was  the  logical  person  to  get  him  [ the 
director  of  the  Center]  that  information.  There  were  other 
people  but  they  had  so  many  other  things  to  do,  or  they  didn't 
really  know  how  to  get  this  kind  of  information.  And  so  it  just 
kind  of  fell  to  us  to  get  it. 

The  newness  and  flexibility  of  the  role  of  the  evaluators  apparently 
reinforces  the  tendency  of  administrators  and  clinicians  in  these 
organizations  to  rely  upon  them  to  carry  out  administrative  and 


data  gathering  tasks. 
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How  important  are  non -evaluative  activities  performed  by  the 

evaluators  in  comparisons  with  evaluative  ones?  There  is  variation 

among  the  organizations  studied  concerning  the  amount  of  time 

devoted  by  the  evaluation  staffs  to  non -evaluative  activities.  In 

at  least  two  of  the  five  organizations  which  had  evaluation  staffs,  it 

appeared  that  these  non-evaluative  duties  are  consuming  a large 

21 

percentage  of  their  time.  Pressman  and  Wildavsky  argue  that  the 

actual  allotment  of  time  to  various  activities  is  the  best  way  of 

finding  out  their  relative  importance: 

If  you  want  to  know  what  matters  most  to  an  organization, 
chart  the  activities  on  which  its  members  spend  their  precious 
allotment  of  hours.  The  allocation  of  time  deserves,  though  it 
does  not  receive,  the  same  attention  that  we  give  to  the 
allocation  of  financial  resources. 

In  Table  4.  7,  the  activities  performed  by  the  evaluation  staff 

2 3 

at  one  Center  are  outlined.  This  description  of  their  activities  is 
based  upon  reports  made  by  the  evaluation  staff  describing  their  own 


21 

This  statement  is  based  on  estimates  of  the  evaluators 
themselves,  interviews  with  clinicians  and  directors  of  the  two 
organizations,  and,  in  one  case,  extended  participant  observation. 

22 

Jeffrey  L.  Pressman  and  Aaron  Wildavsky,  Implementation 
(Berkeley:  University  of  California  Press,  1973),  p.  121. 

23 

The  evaluators  in  this  organization  did  have  formal  responsibility 
for  planning  and  program  development  in  the  organization.  To  a 
certain  extent,  then,  the  focus  on  non-evaluative  activities  is  based 
on  actual  formal  duties.  But  the  extent  and  consistency  to  which  these 
latter  activities  displaced  the  evaluative  ones  is  the  issue  here. 
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duties.  It  substantially  agrees  with  estimates  of  activities  performed 
based  on  the  author's  participant  observation  and  interviews  with  the 
evaluation  staff  and  clinical  personnel  of  the  Center. 

TABLE  4.  7 

ACTIVITIES  PERFORMED  BY  AN  EVALUATION  STAFF 

AT  ONE  CENTER 


(1)  Accounting  Activities,  Billing  Systems,  Accountability 
Forms,  and  Administration  of  State  Laws:  The  evaluation  staff  is 
substantially  involved  in  all  of  the  above  activities.  Each  of  these 
activities  normally  consumes  a substantial  portion  of  the  evaluation 
staff's  time  each  day.  Each  of  the  activities  is  necessary  for  the 
Center  to  obtain  needed  funds  from  clients  and  other  sources. 

(2)  Continuation  Application,  District  Plans,  Foundation  Support, 
and  Other  Plans  and  Grant  Applications:  The  evaluation  staff  spends 
considerable  amounts  of  time  on  each  of  the  above  activities.  The 
continuation  grant  application  and  the  district  plan  both  take  several 
months  to  complete.  The  other  grants  and  plans  carried  out  by  the 
evaluation  staff  also  take  substantial  amounts  of  time  such  as  several 
weeks  to  complete. 

(3)  Operational  and  Comprehensive  Planning:  Members  of  the 
evaluation  staff  are  responsible  for  carrying  out  operational  planning 
in  response  to  problems  that  arise  daily.  Also,  one  of  the  members 
of  the  evaluation  staff  is  assigned  to  carry  out  more  comprehensive, 
future -oriented  planning. 

(4)  Prevalence  Study,  Consumer  Satisfaction  Study,  and  Goal 
Attainment  Scaling  Evaluation:  The  evaluation  staff  has  begun  to 
carry  out  a prevalence  study  of  mental  health  needs  in  the  Center's 
catchment  area,  although  this  study  was  delayed.  At  the  time  of  this 
research  project,  they  were  planning  to  carry  out  consumer 
satisfaction  surveys  and  goal  attainment  evaluation  with  respect 

to  certain  programs  in  the  Center. 
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Many  of  the  activities  performed  by  the  evaluation  staff  are 
purely  administrative  type  of  activities  such  as  those  in  category 
one  of  Table  4.  7.  Activities  such  as  setting  up  billing  systems, 
collecting  data  requested  by  funding  agencies  for  accountability 
purposes,  and  implementing  state  laws  are  duties  which  are  necessary 
if  the  organization  is  to  continue  to  receive  funds  from  the  State  and 
its  clientele.  These  activities  take  up  a large  part  of  the  evaluation 
staff's  time  each  day. 

Likewise,  the  activities  listed  in  category  two  of  Table  4.  7 
are  important  to  the  survival  of  the  organization.  They  have  to 
complete  the  continuation  grant  application  and  the  District  Plan 
if  they  are  to  receive  funding  from  the  national  and  state  agencies 
respectively.  This  type  of  activity  cannot  be  postponed  because 
these  grants  and  plans  have  deadlines  associated  with  them. 

Operational  planning  referred  to  in  category  three  also 
involves  activities  characteristic  of  day-to-day  administration. 

The  evaluation  staff  in  this  Center  did  allocate  specific  individuals 
to  have  responsibility  for  comprehensive  planning  and  evaluation. 
However,  when  deadlines  approach  or  someone  is  needed  to  help 
with  these  administrative  duties,  these  persons  often  dropped  their 
planning  and  evaluation  activities  to  help  perform  these  other  duties. 

Thus,  the  evaluators,  clinicians,  and  director  of  the  Center 


noted  that  the  evaluators  have  little  time  to  focus  on  formal  evaluation. 
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They  are  continually  displaced  from  attempts  to  carry  out  formal 
evaluation  and  directed  toward  the  many  administrative  duties  required 
to  support  the  existence  of  the  organization.  In  contrast  to  the 
administrative  activities,  the  evaluation  activities  carried  out  at  this 
Center  had  no  such  deadlines  associated  with  them.  They  did  not 
promise  any  kind  of  direct  financial  return  to  the  Center  except 
in  cases  where  evaluation  efforts  are  directly  supported  by  funding. 

The  result  of  this  situation  is  that  activities  that  are  not  directly 
associated  with  funding  and  that  have  no  deadlines  or  penalties 
attached  are  constantly  sacrificed  to  more  pressing  demands. 

This  same  kind  of  displacement  from  evaluation  toward 
administration  occurred  in  at  least  one  other  Center.  There  are 
several  potential  explanations  for  this  phenomenon  in  these  two 
Centers.  A version  of  "Gresham's  Law  of  Planning"  states  that 
programmed  activities  tend  to  drive  out  non-programmed  activities.  ^ 
In  part,  this  behavior  may  reflect  the  tendency  of  persons  to  become 

9 c 

attached  to  routines  or  to  respond  only  to  deadlines  and  crises.  3 
It  may  also  reflect  the  deliberate  cooptation  of  the  evaluators  by  the 

24 

See  Herbert  A.  Simon,  The  New  Science  of  Management 
Decision  (New  York:  Harper  and  Row,  1960),  pp.  1-13. 

25_ 

Over-attachment  to  routines  is  one  form  of  what  is  called 
"bureaupathology.  " See  Victor  A.  Thompson,  Modern  Organization 
(New  York:  Alfred  A.  Knopf,  1961). 
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director  and  clinical  staff  of  the  organization  as  the  following  evaluator 
suggests: 


(Evaluator)  It  goes  to  this  whole  business  of  the 
cooptation  of  the  evaluator.  And  I think  in  many  senses 
that  that  might  be  what  is  going  on  around  here:  that  there  is  a 
substantial  fear  that  we  have  the  skills,  that  if  I am  left  enough 
time  to  do  the  job  with  my  staff,  that  we  could  probably  come 
up  with  some  fairly  sophisticated  methods  for  evaluation.  I 
am  not  saying  it  is  overt  but  it  may  be  an  unconscious  feeling 
on  their  part.  . . . 

On  the  other  hand,  another  evaluator  in  the  same  Center  doubts 
the  sincerity  of  the  other  evaluator's  protests  against  the  many  non- 
evaluative  duties  he  performs: 

(Evaluator)  He  [head  of  the  evaluation  staff]  tells  me 
that  his  main  interest  is  research,  that  he  doesn't  like  administration 
and  all  of  that,  but  I know  he  likes  it  or  he  wouldn't  be  doing  it. 

That's  him.  . . . 

One  evaluator  in  the  same  organization  also  noted  that  some  of  the 
members  of  the  evaluation  staff  see  their  major  chance  of  mobility 
as  being  within  the  area  of  administration.  In  short,  personal 
ambitions  and  attachment  to  bureaucratic  routines  may  explain  to 
some  degree  why  evaluators  in  two  Centers  spent  so  little  time 
evaluating. 

A different  interpretation  which  is  supported  by  evidence  from 
participant  observation  and  interviews,  is  that  these  programmed 
activities  are  visibly  associated  with  the  organization's  survival 
and  thus  receive  a higher  priority  than  other  activities,  like  measuring 
the  degree  of  goal  attainment  through  evaluation.  While  evaluation  has 
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certain  benefits  expected  to  be  derived  from  it,  such  as  getting 
a headstart  on  future  requirements,  administrators  in  two  of  these 
Centers  found  it  highly  tempting  to  use  evaluators  for  activities 
which  are  more  directly  supportive  of  the  organization's  survival. 

In  short,  three  conditions  seem  to  make  the  displacement  of 
the  evaluators  away  from  evaluation  activities  highly  likely  in  these 
two  Centers:  (1)  the  fact  that  the  organization  is  perceived  to  be 
in  funding  difficulty  and  concerned  with  its  survival,  (2)  the  commit- 
ment of  the  leadership  and  staff  of  the  organization  to  the  organiza- 
tion's survival,  and  (3)  the  perception  that  evaluation  is  not  as 
directly  supportive  of  the  organization's  survival  as  other  non- 
evaluative  activities.  In  the  organizations  in  which  there  has  been 
less  displacement  of  the  evaluators  from  evaluation,  evaluation 
appears  to  be  more  highly  regarded  because  it  has  brought  in 
funds  through  grants  directly  tied  to  evaluation. 

One  final  fact  from  the  listing  of  the  activities  performed  by 
the  evaluation  staff  in  one  Center  should  be  noted  in  Table  4.  7:  the 

tremendous  amount  of  time  devoted  to  paperwork  required  by  outside 

2 fi 

agencies  and  the  district  board.  ° The  large  amount  of  paperwork 
2 

The  activities  performed  by  the  evaluation  staff  described  in 
Table  4.  7 represent  only  one  case.  However,  most  of  the  same 
activities  are  carried  out  in  all  of  the  other  thirteen  organizations. 
However,  in  some  cases,  the  evaluation  staffs  appeared  to  be  less 
involved  in  the  administrative  duties. 


required  to  be  filled  out  as  part  of  their  jobs  irritated  many  of  the 
clinicians  in  several  organizations  studied.  When  clinicians  were 
asked  what  part  of  their  jobs  they  like  least,  many  replied  that  it 
is  the  paperwork  or  "filling  out  forms  for  the  State.  " One  clinician 
stated  that  he  felt  that  he  and  his  colleagues  are  becoming  a "race 
of  administrators.  " A director  expresses  his  frustration  for  the 
continually  escalating  demands  for  outside  evaluation: 

(Director)  We  have  community  evaluations.  We  have  a 
committee  of  the  medical  society,  for  example,  coming  here 
to  meet  with  our  program  directors,  evaluating  our  pro- 
grams, later  this  afternoon.  Last  week  we  had  a two-day 
evaluation  from  the  State  Division  of  Mental  Health.  They  spent 
two  days  with  us  and  we  will  have  a site  visit  for  our  child 
development  grant.  We  have  all  sorts  of  requests  for  these 
things.  . . . Every  time  we  submit  a grant  to  the  state  or 
federal  government--they  have  a committee  that  comes  in. 

We  are  evaluated  to  the  point  that  we  say,  "Great  God,  here  comes 
somebody  else  to  evaluate  us.  " 

The  paperwork  required  by  the  outside  agencies  also  tends  to 
make  many  of  the  clinical  staff  vent  their  frustration  on  the  inside 
evaluators  who  often  actually  have  to  collect  the  data: 

(Supervisory  Clinician  #1)  I am  not  against  account- 
ability and  research  but  in  the  last  year  paperwork  is  increasing.  . . 
the  whole  thing  about  paperwork- -how  it  has  been  laid  upon  us 
as  a method  of  control. 

(Supervisory  Clinician  #2)  The  categories  on  it  don't 
accurately  reflect  what  I am  doing. 

O o 

(Supervisory  Clinician  #3)  Those  cards  look  more  relevant.  ° 


28 


The  computer  cards  replaced  the  activity  sheets. 
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(Director)  Yes,  the  activity  sheets  are  a method  of 
control  but  is  that  bad  or  wrong? 

(Supervisory  Clinician  #1)  The  amount  of  resources 
devoted  to  research  and  evaluation  is  too  great.  We  need 
evaluation  and  research  but  how  much  do  we  need  them  in 
comparison  with  direct  services? 

As  noted  above,  most  of  the  data  required  by  the  funding  agencies 

and  district  board  are  used  by  these  outside  organizations  to  evaluate 

the  performance  of  the  local  centers.  The  amount  of  effort  required 

to  meet  these  requests  for  outside  evaluation  and  accountability 

appear  in  some  of  the  organizations  studied  to  create  resistance 

to  internal  evaluation.  In  short,  the  acceptance  of  internal  evaluation 

by  the  personnel  of  an  organization  may  be  inversely  related  to  the 

2 Q 

amount  of  evaluation  required  by  outsiders. 


Summary  of  the  costs  and  benefits  of  evaluation 

In  the  large  majority  of  the  organizations  studied  there  is  concern 
with  the  survival  of  the  organization  because  of  current  funding 
difficulties.  Evaluation  usually  takes  up  a significant  amount  of 
resources.  It  may  also  generate  anxiety.  Personal  values  and 
ambitions  are  important  in  the  decision  to  invest  in  formal  evaluation 
and  hire  an  evaluation  staff.  But  from  the  viewpoint  of  the  survival 

2 Q 

On  a few  occasions,  such  data  maybe  used  internally.  In 
at  least  one  case,  they  were  used  by  the  evaluator  and  director  to 
support  their  arguments  concerning  the  way  that  the  clinical  staff 
spent  their  time. 
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of  the  organization,  the  following  categories  appear  to  be  the  major 


costs  and  benefits  (either  actual  or  expected)  associated  with  formal 

30 

organizations  under  study: 


evaluation  in  the  fourteen 

Costs  of  Evaluation 

(1)  Money,  time,  and  resources 
used  by  the  evaluation  staff 


(2)  Time  and  other  resources 
used  by  the  clinical  staff 
members  in  cooperating  with 
the  evaluation 

(3)  The  anxiety,  tensions,  and 
conflict  generated  by  the 
evaluation 

(4)  The  potential  costs  of  a 
negative  evaluation  or  irrelevant 
evaluation  upon  the  prestige  and 
morale  of  the  organization 


Benefits  of  Evaluation 

(1)  Use  of  evaluation  to  anticipate 
future  requirements  to  achieve 
greater  input  and  control  over 
future  evaluation 

(2)  Use  of  evaluation  to  justify 
funds  spent  on  mental  health 
services  and  to  support  requests 
for  more  funds 

(3)  Actual  or  expected  increases 
in  funding,  grants,  and  other 
resources  as  a result  of 
undertaking  formal  evaluation 

(4)  Contributions  of  the 
evaluation  staff  to  non -evaluative 
activities  such  as  administration, 
data  gathering. 


Thus,  interviews  and  participant  observation  support  the  view 
that  pragmatic  concerns  with  survival  tend  to  play  a more  important 
role  in  stimulating  interest  in  formal  evaluation  than  the  idealistic 
functions  chosen  as  most  important  on  the  questionnaires. 


It  is  difficult  to  separate  an  individual's  values  and  personal 
ambitions  from  those  which  he  holds  as  a result  of  the  roles  that 
he  plays  in  the  organization.  See  A.  Paul  Hare,  Handbook  of  Small 
Group  Research  (New  York:  Free  Press,  1962),  p.  8. 
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The  differences  between  the  responses  to  the  questionnaire  and 
the  functions  actually  served  by  evaluation  may  be  due  in  part  to 
the  limitations  of  the  instrument.  However,  they  are  also  likely  to  be 
in  large  part  the  result  of  the  tendency  of  our  ideal  motivations  to  be 
displaced  by  more  pragmatic  concerns --a  variation  of  the  previously 
mentioned  Gresham's  Law  of  Planning.  An  example  of  this  tendency 
occurred  in  an  interview  with  a psychiatric  social  worker  who 
responded  in  the  following  manner  when  asked  to  identify  the  major 
function  of  evaluation: 

(Supervisory  Clinician)  Helping  to  determine  which 
activities  are  helpful  and  produce  results  and  which  ones  haven't 
and  can  thus  be  abandoned. 

But  when  the  same  clinician  was  asked  what  functions  the  evaluators 
in  the  Center  had  actually  performed  with  respect  to  the  program 
she  was  involved  in,  she  replied  that  it  was  in  helping  to  write  up 
grant  applications  so  that  she  didn't  have  to  worry  about  them  or 
have  "those  wild  fancies  myself.  " She,  herself,  noticed  discrepancy 
between  this  description  of  the  duties  performed  by  the  evaluation 
staff  and  what  she  had  indicated  earlier  to  be  the  most  important 
function  of  formal  evaluation.  Commenting  on  this  discrepancy,  she 
states  the  following: 

(Supervisory  Clinician)  If  they  get  more  staff,  I would 
like  to  see  them  help  more  in  determining  what  I referred  to 
earlier,  that  which  is  effective  and  that  which  isn't.  . . . 

In  short,  there  seems  to  be  a disparity  between  idealistic  preferences 

of  the  staff  concerning  the  functions  of  evaluation  and  the  way  that 

it  is  actually  used. 
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Organizational  Differences 

Interviews  and  participant  observation  suggest  that  there  are 
differences  among  organizations  concerning  their  perspective  toward 
formal  evaluation.  Some  of  these  differences  have  already  been 
noted.  In  Chapter  III,  we  saw  that  evaluation  is  closely  related  to 
the  values  of  the  personnel  of  the  organizations.  In  some  organi- 
zations, there  appears  to  be  a fairly  cohesive  set  of  values,  and 
consequently,  a fair  degree  of  agreement  on  how  to  evaluate  the 
services  offered  by  the  organization.  In  other  organizations,  there 
are  sharp  differences  in  the  values  held  by  different  groups  of 
organizational  personnel  making  it  virtually  impossible  to  achieve 
a consensus  on  how  to  evaluate  the  work  performed  by  organizational 
members. 

In  this  chapter,  we  have  already  noted  that  certain  organizations 
appear  to  be  much  more  willing  than  others  to  invest  in  formal 
evaluation.  The  question  naturally  arises  as  to  what  are  the  important 
factors  which  lead  some  organizations  to  be  more  interested  than 
others  in  formal  evaluation.  The  data  used  to  determine  these 
factors  is  basically  "soft"  including  interviews  and  participant 
observation.  Also,  inferences  beyond  the  particular  group  of 
organizations  studied  are  problematical. 

Given  these  limitations,  it  appears  that  the  attitude  of  the 
director  of  the  organization  toward  formal  evaluation  is  a key  factor 
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in  determining  whether  these  organizations  invest  in  evaluation  and 
an  evaluation  staff.  One  evaluator  describes  the  role  of  the  director 
as  follows: 


(Evaluator)  He  [the  director]  was  the  impetus.  His 
first  idea  when  he  came  to  the  clinic  was  to  conduct  a survey 
of  the  needs  of  the  community.  Then  he  started  talking  with  a 
variety  of  people  in  the  State  and  got  into  program  evaluation 
as  a result  of  that.  . . . He  brought  that  back  to  the  staff  who 
said  yes  also. 

A director  who  decided  to  undertake  a consumer  satisfaction  study 
but  not  yet  to  invest  in  an  evaluation  staff  openly  admits  the  crucial 
importance  of  his  own  role: 

(Director)  I think  that  it  is  important  what  the  director 
thinks --where  you  put  your  emphasis.  I don't  think  that 
unless  I put  emphasis  on  it  [evaluation]  that  it  would  be  adherred 
to.  I initiated  the  consumer  satisfaction  study  and  the  initial 
reaction  was  negative.  1 don't  know  how  it  is  going  to  work  out, 
since  I am  obviously  ambivalent  about  evaluation  myself. 

Another  example  of  the  director's  importance  occurred  in  a 

Center  where  the  director  became  interested  in  evaluation  and 

subsequently  began  to  carry  out  some  of  his  own  evaluation  besides 

having  his  Center  associated  with  other  organizations  involved  in 

evaluation.  These  activities  ceased  when  he  subsequently  left  the 

Center  and  a new  director  who  had  no  interest  in  formal  evaluation 

took  over. 

Of  course,  other  factors  can  influence  the  director's  degree  of 
importance,  such  as  his  style  of  administration,  the  extent  of  his 
authority,  and  the  "climate"  of  interpersonal  relationships  in  the 
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organization.  Yet,  even  in  a Center  which  covered  an  enormous 
catchment  area  and  was  supposedly  moving  toward  decentralization, 
the  director  indicated  in  an  interview  with  the  author  that  he  thought 
that  he  could  "pretty  well  take  care  of  the  evaluation  programs,  " 
including  internal  resistance  to  it,  if  he  had  the  support  of  the  district 
board.  In  short,  all  of  the  directors,  clinicians,  and  evaluators 
with  whom  the  author  spoke  agreed  that  strong  support  of  the  director 
of  the  organization  was  crucial  in  order  for  an  evaluation  program 
to  be  established. 

Evidence  concerning  other  potentially  important  factors  is 

t 

more  ambivalent.  The  financial  condition  of  the  organization  might 
affect  attitudes  toward  investing  in  formal  evaluation.  It  would  seem 
natural  for  a "poor"  organization  to  be  less  inclined  to  engage  in 
evaluation  than  a "rich"  organization  with  slack.  However,  the 
possibility  of  getting  grants  might  encourage  a poor  organization  to 
go  into  evaluation.  A rich  organization  would  not  have  this  incentive 
to  the  same  degree.  Therefore,  the  effect  of  the  financial  factor 
is  problematical. 

In  other  words,  the  potential  benefits  of  evaluation  might  seem 
more  impressive  to  a struggling  organization  but  so  would  the  costs, 

31 

See  Fred  E.  Fiedler,  A Theory  of  Leadership  Effectiveness 
(New  York:  McGraw -Hill  Book  Company,  1967). 
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while  the  opposite  would  be  true  of  a more  wealthy  organization. 

Unfortunately,  it  was  impossible  to  collect  "hard"  data  concerning 

the  amount  of  slack  actually  available  in  an  organization.  Indeed, 

it  has  been  argued  that  slack  is  actually  a "psychological"  variable.  ^ 

Regardless  of  how  defined,  subjective  evidence  accumulated  during 

the  interviews  indicated  that  both  the  objective  and  subjective  amount 

of  slack  of  the  organizations  which  invested  heavily  in  evaluation 

roughly  paralleled  that  of  those  organizations  less  committed  to 

evaluation.  Of  course,  nearly  all  of  the  organizations  appeared 

quite  concerned  with  finances.  Another  potentially  important 

variable  is  the  size  of  the  organizations.  On  the  one  hand,  it  might 

be  easier  to  introduce  evaluation  in  a smaller  organization.  On 

the  other  hand,  larger  organizations  are  likely  to  have  more 

34 

resources  to  invest  m such  activities.  The  organizations  most 


Thus  a fiscally  conservative  organization  might  require 
much  more  substantial  reserves  or  "surplus"  than  a more  liberal- 
minded  (fiscally)  organization  before  undertaking  an  extra  activity, 
such  as  formal  evaluation.  This  would  make  it  impossible  to 
determine  the  amount  of  slack  based  only  on  their  financial  status. 

See  Victor  A.  Thompson,  Bureaucracy  and  Innovation,  pp.  43-45. 

33 

The  one  organization  which  appeared  to  have  slack  did  invest 
in  a research  and  evaluation  staff. 

34 

James  Q.  Wilson  argues  that  the  size  of  an  organization  has 
an  ambiguous  effect  on  the  amount  of  innovation  that  takes  place 
in  organizations.  See  James  Q.  Wilson,  'Innovations  in  Organizations : 
Notes  Towards  a Theory,  " in  Approaches  to  Organizational  Design, 
ed.  James  D.  Thompson  (Pittsburgh:  University  of  Pittsburgh 
Press,  1966),  pp.  193-218. 
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heavily  involved  in  evaluation  ranged  from  the  very  small  to  some 
of  substantial  size. 

There  is  one  other  factor  that  interviews  and  participant 
observation  indicated  to  be  of  some  importance  in  determining  the 
extent  of  interest  organizations  have  in  evaluation- -the  nature  of  the 
ambitions  and  values  held  by  important  individuals  in  the  organiza- 
tions. It  has  already  been  noted  that  the  values  and  attitudes 
of  the  director  can  be  important  in  determining  the  nature  of  the 
evaluation  and  degree  of  emphasis  put  on  it.  However,  up  until  this 
point,  the  director's  attitudes  have  been  discussed  mainly  in  terms 
of  his  organizational  role  (e.  g.  , his  goals  for  the  organization  and 
his  interpretation  of  the  costs  and  benefits  that  evaluation  would 
contribute  to  its  survival). 

One  does  not  have  to  be  cynical  to  know  that  personal  ambitions, 

and  not  just  formal  obligation,  play  an  important  part  in  the  behavior 

of  individuals.  There  appeared  to  be  some  correlation  between 

interest  in  investing  in  formal  evaluation  and  achieving  recognition 

35 

beyond  the  local  organization.  For  example,  staff  members  from 
two  of  the  organizations  most  advanced  in  evaluation  had  achieved 

r-jr  " 

uoThis  is  similar  to  the  distinction  made  between  localism  and 
cosmopolitanism  orientations  discussed  in  organizational  and 
sociological  literature.  See  Alvin  W.  Gouldner,  "Cosmopolitans 
and  Locals:  Toward  an  Analysis  of  Latent  Social  Roles- -I,  " 
Administrative  Science  Quarterly,  Vol.  2 (September,  1957), 
pp.  281-306. 
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some  degree  of  recognition  in  the  field  through  delivering  papers 
at  conferences,  having  references  made  to  their  evaluations  in 
publications  in  the  evaluation  field,  and  receiving  numersous  visitors 
to  observe  and  "adopt"  their  "model"  of  evaluation.  A member  of 
the  staff  at  another  Center  admitted  that  he  thought  the  head  of  the 
evaluation  staff  was  thinking  about  publishing  some  survey  research 
they  were  doing.  Another  evaluator  stated  that  evaluation  is  more 
important  than  his  other  duties  because  of  the  potential  recognition 
to  be  achieved  from  it: 

(Evaluator)  Going  back  to  the  original  point  of  view, 
that  my  career  is  ultimately  going  to  be  aggrandized  by  the 
publications  that  I have  got,  and  that  my  future  is  dependent  upon 
my  production  in  that  sense,  the  evaluation  component  is  most 
important. 

In  short,  personal  ambitions,  both  of  the  directors  and  other  key 
members  of  the  staff  of  the  organizations,  can  affect  the  interest 
or  lack  of  interest  in  evaluation,  depending  on  if  these  individuals 
see  evaluation  as  contributing  to  the  accomplishment  of  their 
personal  goals. 

Interest  in  evaluation  also  seems  to  be  positively  related  to  a 
general  interest  in  obtaining  "extra"  grants  for  special  experimental 
programs.  A staff  member  of  an  organization  that  was  doing  little 
about  formal  evaluation  or  securing  grants,  describes  the  reasons 
for  the  position  of  the  Center: 

(Supervisory  Clinician)  Well,  I think  that  most  of  the 
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others'  experience  along  that  line  would  tend  to  make  them, 
in  hindsight,  shy  away  from  that.  We  shy  away  from  getting 
this  six  months  type  of  grant  and  thereafter --find  it  cut  out. 

We  consider  ourselves  adequately  funded  and  our  thrust  is  not 
towards  getting  that  grant  for  epileptics  or  alcoholism.  . . . 

A director  of  another  center  doing  little  in  the  evaluation  field 

stated  that  he  would  accept  the  evaluation  of  a program  at  another 

organization  as  the  equivalent  to  the  evaluation  of  the  same  program 

at  his  own  Center: 

(Director)  My  program  isn't  that  different.  . . Let 
them  evaluate  one  program  in  the  State  and  I will  accept  that 
as  being  applicable  to  mine. 

A clinician  with  supervisory  responsibilities  for  a program 
funded  by  NIMH  in  a Center  that  was  highly  involved  with  formal 
evaluation  also  noted  a difference  between  centers  that  were  "just 
coasting"  and  those  that  "were  doing  the  job.  " However,  he  felt 
that  the  latter  group  were  not  being  adequately  rewarded  for  their 
extra  efforts: 

(Supervisory  Clinician)  The  least  satisfying  thing  is 
having  to  fool  with  a government  that  is  wishy  washy  about  what 
they  want.  They  will  push  for  something  at  one  time  and  get 
you  out  there  with  the  program  and  then  decide  that  the  push 
is  off.  It  leaves  you  out  there  half  in  the  program  and  half 
out.  . . . This  gets  very  frustrating  because  we  have  had  to 
try  to  make  this  [cut  in  grants]  up  with  patient  fees  and  locally 
funded  money  and  we  don't  get  a tremendous  amount  of  patient 
fees.  There  doesn't  seem  to  be  any  reward  for  those  centers 
doing  the  job  as  opposed  to  those  just  coasting. 

To  repeat  our  argument,  the  decision  to  invest  heavily  in 

evaluation  is  affected  by  the  nature  of  the  ambitions  of  the  staff  of 
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the  organizations,  particularly  by  key  personnel  such  as  the  director 
and  program  heads.  There  seem  to  be  differences  among  the 
personnel  of  the  organizations  concerning  the  extent  to  which  they 
possess  ambitions  oriented  toward  achieving  recognition  beyond 
the  confines  of  the  local  area. 

Summary 

The  purpose  of  this  chapter  was  to  explore  the  functions 
that  evaluation  performs  for  the  mental  health  organizations  under 
study.  In  particular,  the  relationship  between  the  organizations' 
struggle  for  existence  and  the  decision  to  undertake  formal  evalua- 
tion was  focused  upon.  The  willingness  of  organizations  to  invest 
in  formal  evaluation  was  particularly  interesting  in  view  of  the  fact 
that,  as  we  saw  earlier,  evaluation  is  not  expected  to  have  much 
of  an  impact  on  organizational  operations.  Despite  the  fact  that 
most  of  these  organizations  are  experiencing  difficulty  in  finding 
the  necessary  funds  to  run  the  organizations  and  meet  salaries, 
many  are  spending  scarce  resources  carrying  out  one  or  more  forms 
of  formal  evaluation.  Is  this  evidence  that  these  organizations, 
even  under  duress,  can  remain  committed  to  high  goal  attainment 
rather  than  to  pragmatic  concerns  of  personal  security  and  organiza- 
tional survival? 


To  explore  this  and  related  issues,  questionnaire  responses 


167 


of  the  staff  of  these  fourteen  organizations  were  examined  concerning 
what  functions  of  evaluation  they  regard  as  most  important.  It 
was  found  that  the  functions  considered  most  important  are  program 
evaluation,  feedback,  efficiency,  and  accountability  to  community. 
There  is  also  a consensus  that  the  least  important  functions  of 
evaluation  are  the  regulation  of  the  behavior  of  the  members  of  the 
organization  and  evaluation  of  individual  performances. 

Two  points  of  particular  interest  were  noted  concerning  the 
above  responses.  The  views  of  the  staff  concerning  the  most 
important  functions  of  evaluation  had  little  relationship  to  the  kinds 
of  evaluation  methods  actually  being  employed  by  the  organizations 
of  which  they  are  members.  Current  evaluation  techniques  are 
actually  more  appropriate  to  evaluation  of  individual  clinicians 
(though  evaluation  is  not  used  this  way)  than  evaluation  of  programs, 
and  to  accountability  to  clientele  than  to  accountability  to  community. 
Thus,  attitudes  concerning  the  relative  importance  of  the  functions 
of  evaluation  are  not  directly  related  to  the  kinds  of  evaluation 
actually  undertaken  by  these  organizations. 

A second  point  of  interest  is  the  fact  that  the  one  function  most 
directly  associated  with  organizations'  survival,  providing  justification 
for  the  funding  of  the  mental  health  organization,  is  rated  low  in 
importance  by  all  groups. 


The  data  show  favorable  attitudes  toward  formal  evaluation 
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as  revealed  in  a series  of  questions  concerning  its  worth.  Likewise, 
the  preferred  functions  are  goal-oriented  rather  than  directed  toward 
such  purposes  as  funding  and  control.  However,  it  was  argued  that 
the  attitudes  concerning  the  most  important  functions  of  evaluation, 
and  also  its  worth,  might  not  accurately  reflect  the  actual  role  it 
plays  in  the  organizations  because  of  limitations  of  the  instrument 
measuring  attitudes  and  also  because  preferences  indicated  on  a 
questionnaire  may  not  predict  behavior. 

Using  data  acquired  from  interviews  at  fouteen  organizations 
and  extended  participant  observation  at  one  Center,  evidence  was 
developed  to  support  the  following  conclusions:  (1)  the  staff  of  these 
organizations  perceive  a crisis  in  funding  which  has  led  to  increased 
potential  for  tension  and  conflict  among  staff  of  different  programs; 

(2)  most  persons  interviewed,  including  directors,  see  concern  for 
survival  and  the  corresponding  outside  pressures  from  different 
levels  of  government  as  the  chief  impetus  behind  the  interest  in 
undertaking  formal  evaluation;  (3)  specifically,  evaluation  is  being 
undertaken  to  a large  degree  to  obtain  some  actual  or  expected 
benefits  to  the  organization  in  its  struggle  for  survival,  such  as 
giving  the  organizations  a headstart  on  requirements  for  evaluation 
likely  to  be  imposed  in  the  future,  justifying  dollars  spent  on  mental 
health  organizations,  obtaining  or  contributing  to  the  acquisition  of 
grants,  and  the  use  of  the  evaluation  staff  for  performing  non -evaluative 


duties . 
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It  is  suggested  that  organizations  such  as  these,  which  are  both 

pressed  for  funding  and  consist  of  highly  educated  individuals,  seem 

to  be  particularly  likely  to  scan  the  environment  for  any  movement 

that  can  be  used  to  advantage.  It  is  also  argued  that  the  man-hours 

of  work  of  the  evaluators  tend  to  be  displaced  away  from  evaluation 

toward  other  activities  that  are  more  closely  associated  with 

organizational  survival,  except  in  those  cases  where  the  organization 

3 6 

had  substantial  slack  or  grants  tied  directly  to  evaluation.  Inter- 
views and  participant  observation  also  show  that  the  personnel  of 
these  organizations  spend  a considerable  amount  of  time  responding 
to  outside  requirements  for  data,  thereby  inhibiting  interest  in 
internal  evaluation. 

Finally,  the  problem  of  why  some  organizations  invested  heavily 
in  formal  evaluation  while  others  did  not  was  investigated.  Evidence 
indicates  that  factors  such  as  organizational  size  and  financial 
status  do  not  explain  the  differences  between  organizations  interested 
in  evaluation  and  those  that  are  not  interested.  Rather,  the  attitudes 
of  the  directors  appear  to  be  the  single  most  important  variable  affecting 

3 6 

In  other  words,  when  the  benefits  from  the  evaluation  are 
more  indirect  such  as  getting  a headstart  on  other  organizations, 
the  evaluator  is  more  likely  to  be  "coopted"  than  when  evaluation 
directly  brings  money  into  the  organization. 
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the  decision  to  invest  heavily  in  formal  evaluation  in  the  group  of 
fourteen  organizations.  The  attitudes  of  the  directors  concerning 
evaluation  appear  to  be  the  result  of  a complex  weighing  and  balancing 
of  the  following  components:  (1)  the  costs  and  benefits  of  evaluation 

in  terms  of  its  impact  on  the  survival  of  the  organization  as  perceived 
by  the  director,  (2)  the  intrinsic  values  and  the  nature  of  the  ambitions 
of  the  director.  The  perspectives,  values,  and  ambitions  of  other 
key  personnel  in  the  organization  may  also  influence  the  decision. 
There  seems  to  be  a relationship  between  interest  in  formal  evaluation 
and  desire  to  achieve  recognition  beyond  the  local  organization. 
Schematically,  the  above  argument  looks  as  follows: 


Concern  for 

organizational 

survival 


Organizational  leadership's 
perception  of  the  costs  and 
benefits  of  evaluation  for 
the  survival  of  the  organi- 
zation 


Organizational  leadership's 
perception  of  the  costs  and 
benefits  of  evaluation  in 
terms  of  their  personal 
values  and  ambitions 


Decision  on  how 
much  to  invest  in 
formal  evaluation, 
if  anything 


Admittedly,  this  model  of  the  decision  process  concerning  investment 
in  formal  evaluation  is  simplified.  There  is  undoubtedly  some 
interaction  between  the  director's  personal  goals  and  his  perception 
of  the  costs  and  benefits  of  evaluation  but  it  appears  that  both  have 
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an  independent  effect  on  the  final  decision. 

Some  of  the  above  conclusions  can  be  put  in  hypothetical  form 
for  future  testing: 

(1)  Evaluation  activities  associated  with  organizational 
survival  will  displace  other  activities  which  are  not  directly 
related  to  organizational  survival. 

(2)  The  degree  of  displacement  of  evaluation  from  goal- 
oriented  objectives  to  survival  objectives  is  a function  of  the 
degree  of  concern  for  organizational  survival  of  organizational 
members  and  the  number  of  demands  made  by  outside  agencies 
on  these  organizations. 

(3)  The  degree  to  which  an  organization  attempts  to 
anticipate  its  environment  is  a function  of  the  interaction  between 
the  degree  of  concern  of  the  organizational  members  for  its 
survival  and  the  extent  to  which  organizational  members  are 
capable  of  scanning  the  environment. 

It  is  now  possible  to  see  that  the  relationship  between  the 
decision  to  undertake  formal  evaluation  and  the  problem  of  organiza- 
tional survival  is  much  more  complex  than  originally  anticipated. 

The  struggle  for  organizational  survival  involves  the  attempt  to 

O O 

secure  the  necessary  resources  to  keep  the  organization  solvent. 

The  struggle  for  organizational  survival  can  indeed  be  an  obstacle 
to  formal  evaluation  in  organizations  because  it  makes  organizational 


o n 

0 iAs  noted  previously,  it  is  difficult  to  sort  out  the  two  factors 
anyway,  although  one  fairly  successful  attempt  is  that  of  Neal  Gross, 
Ward  S.  Mason  and  Alexander  W.  McEachern,  Explorations  in  Role 
Analysis:  Studies  of  the  School  Superintendency  Role  (New  York: 
John  Wiley  and  Sons,  1958). 

38 

For  a description  of  various  strategies  of  organizational 
survival,  see  Chapters  18  and  19  in  Herbert  A.  Simon,  Donald  W. 
Smithburg,  and  Victor  A.  Thompson,  Public  Administration  (New 
York:  Alfred  A.  Knopf,  1950). 
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members,  and  especially  the  leadership,  critically  examine  the 
costs  of  evaluation.  Even  when  evaluators  are  hired,  they  are  often 
likely  to  be  displaced  from  purely  evaluative  activities  to  non- 
evaluative  ones.  However,  ironically,  it  is  also  concern  with 
organizational  survival  that  has  been  the  single  greatest  stimulant 
to  undertake  evaluation  voluntarily  in  the  group  of  organizations  under 
study.  The  reason  for  this  apparent  anomi  ly  is  that  undertaking 
formal  evaluation  is  perceived  to  be  potentially  related  to  certain 
concrete  benefits  for  the  organization  and  also  as  a means  of 
minimizing  costs  of  evaluation  requirements  likely  to  be  imposed 
in  the  future. 

Evaluations  undertaken  for  such  reasons  are  likely  to  reflect 
positively  on  the  organization.  In  an  atmosphere  characterized 
by  tension  and  real  concern  about  organizational  survival,  evaluations 
that  reflect  negatively  on  the  performance  of  personnel,  programs, 
or  the  work  of  the  entire  organization  can  be  expected  to  arouse  a 
great  deal  of  resentment  and  resistance.  Evaluators  have  generally 
minimized  the  potential  costs  of  evaluation  for  organizational  members 
by  introducing  evaluation  in  a non -threatening  way  and  undertaking 
evaluations  that  are  likely  to  be  positive  in  nature. 

Concerning  variation  among  the  organizations  in  their  interest 
in  evaluation,  it  was  found  that  the  differences  can  be  explained  in 
large  degree  by  the  attitudes  of  the  director.  Such  a finding  is 
surprising  because  these  organizations  are  composed  largely  of 
professionals  who  pride  themselves  on  their  individualism  and 
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autonomy.  The  finding  reinforces  the  view  that  formal  hierarchy 
is  still  an  extremely  important  organizational  variable,  even  in 
organizations  which  consist  of  individuals  most  likely  to  resist 
the  arbitrary  use  of  formal  authority. 


CHAPTER  V 


EVALUATION,  ACCOUNTABILITY,  AND  RESPONSIBILITY 


Evaluations  are  often  intended  to  enforce  responsiveness  to 

certain  values.  * The  particular  method  by  which  this  is  accomplished 

can  vary  from  critical  self-evaluations  carried  out  by  concerned 

groups  of  professionals  to  outside  evaluations  performed  by  the 

2 

boards  and  funding  agencies  of  mental  health  organizations.  There 
are  also  differences  in  the  kinds  of  values  which  different  evaluations 
emphasize.  They  range  from  standards  set  by  societies  of  professionals 
to  "popular'1  values  held  by  the  public,  politicians,  and  influential 
groups  in  a community. 

In  Chapter  I,  it  was  pointed  out  that  one  of  the  major  distinctions 
between  most  public  and  private  organizations  is  the  fact  that  public 
organizations  lack  the  "automatic"  accounting  criteria  of  profit. 


■^As  noted  in  Chapter  I,  accountability  is  defined  as  the  methods 
by  which  the  responsiveness  is  achieved  and  responsibility  refers 
to  the  actual  responsiveness.  However,  as  is  discussed  in  this 
chapter,  the  distinction  between  these  two  concepts  is  not  absolute 
because  of  the  phenomenon  of  goal  displacement. 

2 

By  outsider  evaluation,  the  author  is  referring  to  any  evaluation 
not  being  carried  out  by  personnel  belonging  to  the  local  organization. 
Some  groups  of  outsiders,  such  as  private  psychiatrists  with  whom 
the  author  talked,  saw  evaluations  by  the  state  and  national  agencies 
as  still  being  "insider"  evaluations  and  not  truly  critical.  This 
problem  is  discussed  further  in  this  chapter. 
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rate  of  return  and  other  indices  of  effectiveness  which  are  available 
to  private,  profit-making  organizations.  Thus,  a tremendous  body 
of  literature  has  developed  in  the  attempt  to  provide  public  organiza- 
tions with  the  equivalent  of  the  automatic  mechanisms  of  the  private 
market.  ^ In  this  sense,  formal  evaluation  may  be  seen  as  the  attempt 
to  bridge  the  gap  between  the  survival  and  goal  attainment  models 
of  organizations  by  making  the  organization's  survival  dependent 
upon  its  performance  and  responsiveness  to  the  values  of  those  who 
can  affect  its  survival. 

As  noted  earlier,  much  of  the  literature  on  evaluation  assumes 

that  only  evaluations  by  outsiders  can  provide  a critical  and  objective 

assessment  of  an  organization's  services.  Nevertheless,  there  are 

important  obstacles  to  the  effectiveness  of  outside  evaluations  as  a 

5 

means  of  achieving  the  desired  responsiveness.  In  this  chapter, 
the  relationship  between  organizational  evaluation  and  the  issues  of 
accountability  and  responsibility  is  analyzed. 


3 

As  noted  earlier,  some  economists  challenge  the  extent  to 
which  automatic  evaluation  actually  takes  place  in  private  organizations. 

^Thus,  cults  of  efficiency  have  engulfed  at  one  time  almost  every 
form  of  public  organization.  See,  e.  g.  , R.  E.  Callahan,  Education 
and  the  Cult  of  Efficiency  (Chicago:  University  of  Chicago  Press,  1962) 
and  Carol  Taylor,  In  Horizontal  Orbit:  Hospitals  and  the  Cult  of 
Efficiency  (New  York:  Holt,  Rhinehart,  and  Winston,  1970). 

5 

For  a general  discussion  of  this  point,  see  Carol  H.  Weiss, 
Evaluation  Research:  Methods  for  Assessing  Program  Effectiveness 
(Englewood  Cliffs:  Prentice  Hall,  1972),  pp.  106-123. 
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Compar  ons  are  utilized  between  internal  and  external 
evaluations  to  give  perspective  to  the  problem  of  achieving 
responsibility.  The  focus  of  this  dissertation  is  internal  organiza- 
tional evaluation.  However,  it  is  useful  to  develop  the  comparison 
between  internal  and  external  evaluations  because  up  until  now  the 
internal  evaluation  of  these  organizations  has  been  compared  with 

O 

abstract  models  of  "critical"  and  "token"  evaluations.  Here,  the 
extent  to  which  internal  and  external  evaluations  are  capable  of 
providing  responsiveness  to  values  other  than  the  sheer  survival 
of  the  organization  is  systematically  examined. 

In  particular,  four  general  problem  areas  are  examined  which 
are  important  for  determining  the  degree  of  responsiveness  ultimately 
achieved  by  an  evaluation:  (1)  the  kinds  of  values  held  by  the 
evaluators  and  the  persons  being  evaluated;  (2)  the  motivations  behind 
the  evaluation- -their  nature  and  strength;  (3)  the  kinds  of  authority 
possessed  and  exercised  by  the  evaluators  and  the  reaction  of  those 
being  evaluated  to  them;  (4)  the  degree  to  which  the  evaluators 
possess  the  information  necessary  to  ascertain  whether  the  evaluation 


Thus,  the  comparisons  and  the  criteria  by  which  the  impact 
of  an  activity  such  as  internal  evaluation  is  judged  may  in  large 
part  determine  the  conclusions  reached.  This  problem  makes  it 
useful  to  compare  "concrete"  internal  evaluation  with  actual,  on- 
going external  evaluation  as  another  point  of  comparison  to  give 
perspective  to  the  study. 
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is,  in  fact,  accomplishing  the  desired  responsiveness. 

Values 

The  values  held  by  the  personnel  of  the  mental  health  organiza- 
tions, district  boards,  and  state  and  national  agencies  are  of 
tremendous  significance  for  the  relationship  among  evaluation, 
accountability,  and  responsibility.  If  the  groups  and  organizations 
demanding  and  responding  to  demands  for  accountability  and  responsi- 
bility hold  similar  values,  then  the  need  for  a critical  outsider 

evaluation  is  likely  to  be  less  than  if  they  hold  widely  divergent 
7 

values.  In  this  section  of  the  chapter,  the  major  values  emphasized 
by  the  centers,  boards,  and  agencies,  and  the  degree  of  congruence 
among  these  values,  are  examined.  A related  issue  is  the  priority 
assigned  to  different  goals.  Even  if  common  goals  can  be  agreed 
upon,  there  can  be  substantial  disagreement  concerning  the  relative 

O 

importance  of  the  different  goals.  A final  problem  is  whether 
clear,  consistent,  and  measurable  goals  exist  which  would  enable 


7 

Herbert  Kaufman  has  made  the  point  that  early  socialization 
to  the  values  of  the  central  agency  often  can  make  formal  control 
devices  unnecessary.  See  Herbert  Kaufman,  The  Forest  Ranger 
(Baltimore:  The  Johns  Hopkins  University  Press,  1969),  pp.  198  and 
passim. 

g 

David  Mechanic  has  pointed  out  the  problem  of  choosing  between 
the  criteria  of  "need"  (who  is  in  the  worst  condition?  " and  "gain" 

(who  is  most  likely  to  gain  from  help?)  in  the  allocation  of  services. 
See  David  Mechanic,  Public  Expectations  and  Health  Care  (New 
York:  John  Wiley  and  Sons,  1972),  pp.  227-231. 
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an  evaluator  within  or  outside  the  organization  to  accurately  measure 
the  degree  of  responsiveness  achieved. 


Values  Within  the  Organizations 

It  was  pointed  out  in  Chapter  III  that  there  is  often  substantial 
disagreement  in  these  organizations  concerning  the  degree  of 
emphasis  to  be  placed  on  different  kinds  of  services,  clients, 
philosophies  of  treatment,  and  so  forth.  The  basic  points  of  agree- 
ment usually  related  to  very  broad  goals  such  as  commitment  to 
improve  mental  health  and  decrease  mental  illness.  9 Beyond 
extremely  broad  goals  such  as  these,  there  is  substantial  dis- 
agreement existing  within  the  mental  health  organizations  concerning 
many  important  issues. 

An  example  of  such  disagreement  concerning  values,  orientations, 
and  priorities  occurred  in  the  following  discussion  in  one  Center: 

(Director)  In  response  to  a query  by  [name  of  a clinician 
interested  in  establishing  mental  health  programs  for  children] 
on  the  problem  of  priorities  for  allocation  of  resources,  the 
following  are  the  top  priorities:  (1)  the  acutely  psychotic; 

(2)  those  on  medicine.  ...  If  the  board  is  not  concerned  with 
children,  then  you  should  see  if  the  citizen's  advisory  groups 
can  be  stirred  up  to  demand  services  for  children. 

(Supervisory  Clinician  #1)  I have  real  reservations  about 
delivering  services  to  children 

(Supervisory  Clinician  #2)  What  will  we  cut  out  and  will  we 


g 

One  evaluator  related  how  he  had  asked  one  program  head  what 
the  goals  of  his  program  were  and  received  the  reply,  "to  eliminate 
mental  illness  in  the  county.  " 
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spread  too  thin  if  we  try  to  provide  services  for  children? 

Later,  in  the  same  discussion,  the  following  exchanges  took  place: 

(Supervisory  Clinician  #3)  I would  like  to  change  the  name 
of  the  Center  to  the  Community  Counselling  Service. 

(Director)  We  are  not  really  a counselling  service.  The 
implication  would  be  that  we  were  limited,  and  also  organiza- 
tions that  are  more  directly  related  to  medicine  can  get  more 
funds . 

(Planner)  But  the  present  name  associates  us  with  the  State 
Mental  Hospital  and  that  keeps  a lot  of  people  from  coming  in. 

(Supervisory  Clinician  #1)  Is  the  question  of  the  name  of  the 
Center  a priority  issue? 

(Planner)  It  is  if  the  name  is  keeping  people  away. 

(Psychiatrist)  It  is  clear  to  me  that  it  is  a question  of  "medical" 
versus  "non-medical"  orientations.  . . . 

(Supervisory  Clinician  #3)  The  real  issue  is  what  the  Center 
is  and  what  it  is  doing. 

(Psychiatrist)  The  real  issue  is  how  we  feel  about  ourselves 
and  the  work  we  are  in. 

(Clinician)  I feel  bad  about  what  we  are  doing  a lot  of  the 
time.  . . . 

(Psychiatrist)  I have  been  in  psychiatry  for  five  years  and 
in  the  last  five,  I haven't  cured  anybody.  I can  define  cure  with 
pneumonia  but  I can't  here. 

(Director)  I feel  much  better  in  dealing  with  the  severely 
disturbed.  You  can  do  something  for  them  quickly.  It  takes 
much  longer  to  help  neurotics. 

The  discussion  went  on  to  other  issues --for  example,  whether  or 
not  it  was  proper  for  mental  health  personnel  to  perform  such  tasks 
as  finding  jobs  for  clients.  Some  felt  that  it  was,  because  other 
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bureaucratic  agencies  in  the  area  had  failed  to  do  so,  but  others 
believed  that  it  would  be  getting  far  afield  from  the  major  goals  of 
the  Center.  The  point  of  the  above  discussion  is  that  there  is  no 
consensus  as  to  the  goals  of  many  of  these  organizations  once  one 
gets  beyond  extremely  broad  generalizations. 

The  lack  of  any  set  of  clearly  defined  goals  and  priorities  has 
important  consequences  for  internal  and  external  evaluations.  It 
increases  the  chance  that  the  clinical  staff  will  overemphasize  the 
particular  indices  measured  by  the  evaluations.  Table  5.  1 shows 
that  large  percentages  of  clinicians,  directors,  and  evaluators 
foresee  the  likelihood  of  goal  displacement  resulting  from  evalua- 
tion. 10  The  following  program  head  describes  the  effect  of  the 
emphasis  of  both  internal  and  external  evaluations  upon  the  number 
of  hours  spent  in  the  direct  delivery  of  services: 

(Supervisory  Clinician)  It  is  like  the  example  of  the 
teacher  who  knows  that  her  children  are  going  to  take  a standard 
achievement  test  and  finds  out  what  is  on  the  test  and  teaches 
for  the  test.  Now  I know  that  our  program  is  being  evaluated 
in  terms  of  how  many  hours  we  spend  in  therapy,  and  so  I have 
been  telling  the  staff,  "You  have  to  get  the  caseload  up  and  get 
these  people  in  for  therapy.  " So  it  is  a controlling  factor. 


^As  used  here,  goal  displacement  refers  to  a certain  kind  of 
unintended  consequence.  It  occurs  when  the  person  being  evaluated 
becomes  over-attached  to  the  particular  form  of  accounting  and 
loses  sight  of  the  more  general  values  (or  overemphasizes  one  value) 
to  which  the  evaluator  wishes  him  to  be  responsive.  In  the  above 
example,  increasing  the  priority  given  to  direct  services  is  intended 
by  the  evaluators.  However,  if  it  were  accomplished  by  so  speeding 
up  the  therapy  so  as  to  result  in  shallow  and  ineffective  treatment, 
it  is  likely  to  be  regarded  as  goal  displacement. 
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TABLE  5.  1 

GOAL  DISPLACEMENT 


Staff  are  likely  to  overemphasize 
evaluation. 

the  activities  measured  by 

Response 

Percentage  of 
Clinicians 

Percentage  of 
Directors  * 

Percentage  of 
Evaluators* 

Strongly  Agree 

5.  3 (2) 

6.  3 (1) 

— 

Agree 

31. 6 (12) 

18.  8 (3) 

33. 3 (3) 

Undecided 

36.  8 (14) 

50.  0 (8) 

44.  4 (4) 

Disagree 

26.3  (10) 

25.  0 (4) 

11. 1 (1) 

Strongly  Disagree 

— 

— 

ll; 1 (l)) 

*Sum  of  percentage  figures  is  unequal  to  100  percent  because  of 
rounding. 


Another  likely  consequence  is  that  the  values  emphasized 
depend  to  a great  degree  on  who  has  control  over  the  evaluations 
of  personnel,  programs,  and  organizations.  Tables  5.2  and  5.  3 
demonstrate  that  a majority  of  clinicians  agree  that  evaluation  is 
likely  to  be  used  as  a means  of  controlling  and  regulating  the  staff 
of  the  organizations. 

Values  of  "Outsiders" 

The  district  boards 

The  major  points  of  disagreement  among  the  personnel  of  the 
mental  health  centers  often  revolve  around  such  issues  as  the  best 
methods  of  treatment  and  priorities  in  the  kinds  of  services  needed. 
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TABLE  5.  2 

EVALUATION  AND  CONTROL:  #1 


Evaluation  is  likely  to  be  used  as  a means  of  control  over  the 
staff. 


Response 

Percentage  of 
Clinicians* 

Percentage  of 
Directors  * 

Percentage  of 
Evaluators* 

Strongly  Agree 

10.  5 (4) 

6.  3 (1) 

— 

Agree 

36.  8 (14) 

18.  8 (3) 

44.  4 (4) 

Undecided 

26.  3 (10) 

43. 8 (7) 

22. 2 (2) 

Disagree 

23. 7 (9) 

31. 3 (5) 

33. 3 (3) 

Strongly  Disagree 

2.  6 (1) 

— 

— 

*Sum  of  percentage  figures  is  unequal  to  100  percent  because  of 
rounding. 

TABLE  5.  3 

EVALUATION  AND  CONTROL:  #2 

Evaluation  is 
clinical  staff. 

likely  to  be  used  to  regulate  the  behavior  of  the 

Percentage  of  Percentage  of  Percentage  of 

Response 

Clinicians  * 

Directors* 

Evaluators* 

Strongly  Agree 

7.  9 (3) 

— 

— 

Agree 

50.  0 (19) 

25.  0 (4) 

22. 2 (2) 

Undecided 

21. 1 (8) 

43.  8 (7) 

11. 1 (2) 

Disagree 

15.  8 (6) 

31.  3 (5) 

55.  5 (5) 

Strongly  Disagree 

5.  3 (2) 

— 

11. 1 (1) 

*Sum  of  percentage  figures  is  unequal  to  100  percent  because  of 
rounding. 
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The  mental  health  district  boards  are  not  perceived  to  be  concerned 
with  the  same  issues.  Rather,  they  are  seen  as  being  interested  in 
the  following  concerns:  (1)  "dollars  and  cents"  issues  such  as  whether 
they  are  getting  their  "money's  worth"  of  service,  (2)  whether  they 
are  being  offered  the  full  variety  of  programs  and  services  that  are 
potentially  available,  (3)  the  status  of  certain  prominent  statistics 
such  as  the  quantity  of  clients  served,  changes  in  inpatient  and  mental 
hospital  loads,  and  similar  statistics.  The  interests  of  the  district 
boards  are  characterized  by  persons  in  the  mental  health  fields  as 
follows: 

(Director  #1)  It  [the  district  board]  is  saying,  "What  are 
the  services  being  offered?  " and  that  is  a very  valid  question. 

It  is  the  equivalent  in  business  of  the  profit  for  the  year.  If 
you  want  to  call  it  evaluation,  you  can,  and  I think  that  the 
board  should  evaluate  that. 

(Director  #2)  We  had  preliminary  hearings  for  the  county 
a week  ago  and  that  is  one  of  the  things  they  are  asking,  "What 
are  you  doing  for  the  money  we  are  giving  you?"  and  "How 
many  people  do  you  serve?  " and  "After  you  serve  them,  how 
many  are  back  on  their  jobs?  " and  "How  many  are  in  the  state 
hospitals  ? " 

(State  Official)  The  boards  feel  a tremendous  responsibility 
for  the  expenditure  of  funds  and  don't  know  where  to  get  the 
information  to  answer  the  questions.  They  are  feeling  politically 
weak. 

While  people  within  the  organizations  see  a potential  overemphasis 
on  questions  of  money  and  quantity,  nevertheless  these  values  are 
recognized  by  most  organizational  members  as  being  legitimate 
concerns  to  which  they  should  be  responsive. 
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Other  values  held  by  members  of  the  district  boards  and 
influential  members  of  the  communities  in  which  they  are  located 
are  actively  opposed  by  some  members  of  the  mental  health  organiza- 
tions. One  psychiatrist  cynically  observed  that  if  their  organization 
was  to  get  more  support  from  the  community  for  its  programs,  it 
would  have  to  show  that  better  mental  health  would  lead  rural  farm 
workers  "to  pick  more  beans.  " The  director  of  a Center  describes 
the  relationship  between  his  own  values  and  those  of  the  community: 

(Director)  Again,  every  community  is  different.  One  of 
the  dominant  values  of  this  community  is  a bias  against  negroes. 

I disagree  with  this  and  have  to  try  to  change  it,  which  is  a 
very  difficult  and  long-term  thing.  Also,  some  people  don't 
want  to  or  aren't  particularly  interested  in  curing  alcoholism 
because  then  they  don't  think  that  they  would  have  the  people 
to  pick  the  oranges.  The  point  is  that  the  community  often 
evaluates  services  according  to  their  own  values,  which  I 
myself  often  disagree  with. 

Some  members  of  mental  health  organizations  are  even  skeptical 
concerning  whether  the  communities  are  truly  interested  in  thrift 
and  efficiency  of  services  delivered.  The  director  of  programs  in  a 
certain  area  thinks  that  the  community's  "evaluation"  of  the  organiza- 
tion is  based  upon  the  amount  of  money  and  jobs  brought  into  the 
community  by  the  Center.  He  expects  there  to  be  a likelihood  of  a 
loss  of  support  for  the  Center  when  the  contributions  from  the  state 
and  national  government  decrease.  In  the  following  discussion  a 
psychiatrist  argues  that  sheriffs  in  the  local  communities  would  use 
a new  law  on  alcoholism  to  their  own  advantage  with  a resulting 
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burden  for  the  Center.  11 

(Supervisory  Clinician)  Won't  the  sheriffs  know  who  to 
admit  and  who  not  to  admit? 

(Psychiatrist)  The  sheriffs  will  have  the  incentive  to  put 
everyone  in  the  hospital  and  keep  them  there  and  let  the  Center 
pay  for  it. 

In  short,  there  are  many  instances  in  which  the  values  of  the 
personnel  of  the  mental  health  organizations  may  conflict  with  the 
values  of  the  members  of  the  district  boards  or  influential  persons 
in  the  communities.  In  such  cases,  internal  evaluation  is  not  likely 
to  make  the  organizations  responsive  to  the  values  of  the  communities. 

Indeed,  the  values  encouraged  by  internal  evaluation  may  be  directly 

1 2 

opposed  to  those  desired  by  the  outside  community. 


The  agencies 

Generally,  the  values  of  the  personnel  of  the  state  and  national 
agencies  are  perceived  to  be  relatively  similar  to  those  of  the 
members  of  the  local  mental  health  organizations.  Yet,  the  state 
and  national  laws  concerning  mental  health  do  not  clearly  specify 


**The  law  requires  that  the  alcoholics  be  treated  rather  than  jailed. 

12 

An  interesting  example  of  the  ambivalence  of  many  people  toward 
the  issue  of  "community  participation"  and  control  occurred  recently 
when  members  of  the  same  group  that  had  encouraged  greater  local 
participation  in  mental  health  centers  took  a position  against  having 
a review  committee  concerning  sterilization  because  it  would  be 
"too  subject  to  local  influence.  " See  Judith  Coburn,  "Sterilization 
Regulations:  Debate  Not  Quelled  by  HEW  Documents,  " Science, 

March  8,  1974,  pp.  935-938. 
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operational  goals  by  which  the  effectiveness  of  mental  health  services 
can  be  measured.  Two  directors  describe  the  limitations  of  the 


state  laws  as  far  as  determining  quality  and  efficiency  are  concerned: 


(Director  #1)  Well,  in  this  instance,  it  [an  evaluation] 
was  just  to  see  if  the  law  was  being  implemented  or  not--whether 
the  services  were  being  provided.  The  Baker  Act  says 
nothing  about  the  quality  of  service.  ^ 


(Director  #2)  What  has  happened  is  that  they  [the  state 
agency]  won't  even  let  us  charge  for  our  own  psychiatrist.  A 
private  psychiatrist  can  be  charged  for  and  it  makes  no  difference 
to  the  state.  They  will  fund  him  because  it  meets  their  criteria 
of  funding,  even  if  it  costs  more. 


The  problem  is  that  the  statutes  concerning  mental  health  do 
not  define  quality,  effectiveness,  or  efficiency.  They  are  mainly 
concerned  with  "intermediate  goals"  such  as  "integration  of  services,  " 
"a  unified  mental  health  system,  " 'participation  by  local  governments,  " 
and  the  "establishment  of  a uniform  ratio  of  state  and  local  participa- 
tion. In  the  absence  of  clearly  defined  "end  goals,  " such 


intermediate  goals  tend  to  become  ends  in  themselves.  Thus  integra- 


tion of  services  or  uniformity  of  ratios  of  participation  are  likely 


to  be  rewarded  whether  or  not  they  actually  lead  to  increased 
effectiveness  of  the  services  offered. 


■^The  Baker  Act  is  the  name  applied  to  legislation  related  to 
the  funding  of  mental  health  centers  in  Florida. 

■^See  Florida,  Florida  Statutes  (1971),  c.  394.66. 
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Much  the  same  is  true  with  respect  to  the  national  agency 
(NIMH)  and  its  goals.  At  its  inception,  three  especially  important 
priorities  characterized  the  movement  toward  community  mental 
health:  (1)  an  emphasis  on  local  treatment  with  a related  decrease 
in  the  size  and  importance  of  large  state  mental  hospitals,  (2) 
emphasis  on  comprehensiveness  and  continuity  of  care  in  the  programs 
and  services  being  offered,  (3)  much  more  emphasis  on  prevention 
through  such  services  as  education  and  consultation. 

While  none  of  these  priorities  has  been  abandoned,  the  limitations 
of  each  have  become  apparent.  Emphasis  on  treatment  in  the 
community  and  decreased  size  of  mental  hospitals  can  greatly 
increase  the  recidivism  rate  and  often  put  a strain  on  the  client's 
family. 15  In  one  Center,  a clinician  suggests  that  the  refusal  of  the 
head  of  the  inpatient  program  to  accept  many  of  his  patients  prevented 
the  personnel  from  spending  adequate  time  on  services  other  than 
aftercare: 

(Supervisory  Clinician)  Our  figures  on  inpatient  care 
may  look  great,  but  we  have  to  spend  50  percent  of  our  time 
on  aftercare  [for  the  patients  whom  they  wished  to  have 
admitted  into  inpatient  care]. 


1 5 

A long  and  continuing  debate  has  occurred  concerning  the 
relative  costs  and  advantages  of  maintaining  patients  in  the  community. 
See,  e.g.  , George  Fairweather  et  al.  , Community  Life  for  the  Mentally 
111:  An  Alternative  to  Institutional  Care  (Chicago:  Aldine  Publishing 
Company,  1969),  pp.  3-7  and  Mervyn  Susser,  Community  Psychiatry 
(New  York:  Random  House,  1968),  pp.  16-17. 
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Some  directors  question  the  goal  of  sharply  decreasing  state 
mental  hospital  patient  loads: 

(Director  #1)  It  is  cheaper  to  keep  him  in  the  State  Hospital 
than  in  the  communities  because  the  communities  don't  have 
any  facilities  for  them  and  there  is  no  more  shining  example  than 
[name  of  the  location  of  the  Center],  We  are  absolutely  swamped. 
We  couldn't  begin  to  handle  people  from  our  area  at  [who  are 
presently  at]  the  State  Hospital.  It  is  a damn  good  political 
argument  to  talk  about  getting  them  back  in  to  the  community 
but  what  are  you  going  to  do  with  them  once  you  get  them  there? 

(Director  #2)  I don't  believe  in  this  nonsense  of  phasing  out 
state  hospitals  and  creating  small  state  hospitals  at  the  community 
level.  I think  that  that  is  stupid.  Some  patients  are  better  off 
at  the  State  Hospital. 

Even  the  concept  of  continuity  of  care  has  been  challenged  as  being 

1 6 

irrelevant  if  the  quality  of  care  is  not  good. 

Although  it  was  the  national  agency  (NIMH)  which  had  strongly 
emphasized  the  need  to  deal  with  prevention,  its  most  recent  forms 

t 

of  "accountability  ."  have  rewarded  institutions  that  have  a large 
number  of  direct  encounters.  At  a meeting  attended  by  local  mental 
health  center  personnel,  an  official  from  NIMH  humorously  referred 
to  the  latest  form  required  to  be  filled  out  by  the  local  organizations. 

He  admitted  the  new  form  would  make  it  seem  as  though  the  entire 
emphasis  of  NIMH  was  changing  away  from  prevention  and  indirect 


*®Thus,  at  a meeting  attended  by  the  author,  an  NIMH  official 
argued  that  it  was  possible  that  a "discontinuity  of  treatment"  might 
be  beneficial  in  certain  cases  if  the  client  has  been  receiving  in- 
appropriate treatment. 
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services: 

(NIMH  Official)  You  are  going  to  be  getting  something  from 
our  office.  It  is  a form  and  it  has  to  be  filled  out.  You  are  going 
to  feel- -and  this  is  true- -that  we  are  changing  our  entire 
emphasis.  The  cost  per  client  will  be  asked  for. 

According  to  some  directors  who  received  the  form,  the  cost  per 

patient  is  defined  in  such  a way  as  to  exclude  indirect  services. 

Summary  of  Values 

Beyond  very  general  goals,  there  is  a lack  of  clearly  defined 
measures  of  the  quality  and  efficiency  of  services  delivered  by 
mental  health  organizations.  As  a consequence,  there  is  a distinct 
danger  of  goal  displacement  due  to  the  overemphasis  upon  the 
particular  criteria  stressed  in  evaluations.  The  values  and  priorities 
of  those  highest  in  the  hierarchy  also  tend  to  be  emphasized  at  the 
expense  of  the  values  and  priorities  of  those  with  less  power. 

District  boards  tend  to  be  interested  in  a few  highly  visible 
criteria  by  which  they  judge  the  performance  of  mental  health 
centers.  These  criteria  include  "dollars  and  cents  items,  " certain 
quantitative  indices,  and  the  variety  of  services  being  offered  by 
the  mental  health  centers.  Members  of  the  mental  health  centers 
(including  its  evaluators)  sometimes  disagree  with  the  certain  values 
held  by  the  boards  and  do  not  wish  to  be  responsive  to  them.  In 
cases  when  internal  and  external  value  systems  diverge  greatly, 
internal  and  external  evaluations  may  come  to  different  conclusions 
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concerning  the  performance  of  the  mental  health  centers. 

The  state  and  national  agencies  also  lack  clearly  defined  goals 
and  priorities.  Personnel  of  the  local  centers  examine  carefully 
the  particular  criteria  of  funding  and  the  nature  of  the  data  requested 
by  these  agencies  in  order  to  have  an  indication  of  what  values  and 
priorities  are  in  current  favor  and  likely  to  be  rewarded. 

The  degree  of  responsiveness  which  an  evaluator  (internal 

or  external)  is  able  to  achieve  is  affected  by  the  following  factors 

related  to  the  issues  discussed  above:  (1)  the  extent  to  which  the 

"end"  goals  and  priorities  favored  by  the  evaluator  can  be  clearly 

1 7 

defined  and  measured,  (2)  the  extent  to  which  the  persons  being 
evaluated  agree  with  the  goals  and  priorities  of  the  evaluator,  (3) 
the  amount  of  unanticipated  displacement  that  occurs  as  a result  of 
the  evaluation.  Thus,  an  evaluator  is  likely  to  achieve  the  kinds  of 
responsiveness  he  desires  if  he  can  clearly  measure  his  goals,  if 
the  people  he  is  evaluating  agree  with  these  values,  and  if  he  is 
able  to  design  an  evaluative  instrument  that  avoids  the  problem  of 
goal  displacement. 


1 7 

The  reasons  for  the  lack  of  clarity  of  "end  goals"  may  result 
from  several  factors.  One  such  factor  is  simply  an  ignorance  or 
uncertainty  about  what  goals  should  be.  Another  argument  is  that  the 
obscurity  of  goals  may  be  the  result  of  compromises  among  different 
individuals  or  groups  in  an  organization.  For  the  latter  point  of  view, 
see  Charles  Lindblom,  "The  Science  of  Muddling  Through,  " Public 
Administration  Review,  Vol.  19  (Spring,  1959),  pp.  79-88. 
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Motivation 

The  Motivation  of  "Outsiders" 


People  may  have  similar  value  systems  but  differ  greatly 
in  the  extent  to  which  they  are  motivated  to  achieve  responsiveness 
to  these  values.  Persons  outside  of  the  organizations  do  not  have 
to  worry  about  internal  concerns  with  organizational  survival  but 
can  afford  to  be  highly  critical  in  their  evaluations.  Indeed,  it  has 
been  argued  that  persons  evaluating  from  the  outside  are  "null 
hypothesis  oriented"  and  eager  to  discover  faults  or  problems  in  the 
services  they  are  evaluating: 

The  independent  evaluator  has  a vested  interest  in  showing 
that  there  was  a need  for  the  evaluation  in  the  first  place  and 
that  the  cost  of  the  evaluation  was  justified  by  the  savings  that 
can  be  made  in  the  program.  Also,  the  evaluator  is  aware 
of  the  greater  drama  and  newsworthiness  in  findings  that  start 
with,  "Surprisingly,  the  program  results  are  the  exactly 
opposite  of  -that  sought.  "18 

As  was  noted  in  the  last  chapter,  there  is  a general  perception 
among  the  directors  of  mental  health  organizations  of  an  increased 
tendency  of  district  boards  and  funding  agencies  to  demand  proof 
of  the  worth  of  mental  health  services.  In  fact,  many  of  the  personnel 
perceive  what  they  believe  to  be  a constant  mistrust  and  pressure  from 


l^Selma  J.  Mushkin,  "Evaluations:  Use  with  Caution,  " Evaluation 
Vol.  1,  No.  2 (1973),  p.  34. 
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these  "outsiders,  " as  does  the  following  director: 

(Director)  There  is  a general  orientation  that  we  are  trying 
to  hide  something  from  them. 

A state  official  who  is  unusually  sympathetic  to  the  situation  of  the 
local  mental  health  organizations  describes  how  most  people  working 
in  the  state  agencies  have  a general  outlook  of  attempting  to  catch 
the  "cheaters": 

(State  Official)  People  who  are  out  to  get  the  cheaters  are 
always  around  and  they  tend  to  win  the  arguments  [in  the  state 
agency]  in  the  long  run. 

However,  there  are  constraints  on  the  extent  to  which  outsiders 
are  motivated  to  evaluate  the  way  that  local  centers  are  using  public 
funds.  Much  of  the  interest  of  members  of  the  communities  in 
evaluation  is  seen  by  the  personnel  in  the  organizations  as  being 
inspired  by  the  political  benefits  which  they  think  can  be  derived 
from  it. 19  Carrying  out  an  evaluation  takes  time  and  money  which 
many  members  of  the  community  (including  members  of  the  district 
boards)  are  not  willing  or  able  to  invest.  20  However,  the  evalua- 
tions which  they  do  carry  out  are  of  a potentially  critical  (and  negative) 


19 Political  and  professional  interests  and  approaches  to  health 
and  mental  health  tend  to  be  quite  different.  See,  e.  g.  , Herbert 
Kaufman,  "The  Politics  of  Health  Planning,  " American  Journal  of 
Public  Health,  Vol.  59  (May,  1965),  pp.  795-796. 

20 

Members  of  other  mental  health  organizations  have  noted  the 
negative  impact  of  the  costs  of  participation  on  community  involvement. 
See,  e.g.  , Seymour  R.  Kaplan  and  Melvin  Roman,  The  Organization 
and  Delivery  of  Mental  Health  Services  in  the  Ghetto  (New  York: 
Praeger,  1973),  p.  104. 
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nature  because  they  are  not  limited  by  the  survival  concerns  of  the 
organization.  ^ 


The  Motivation  of  Insiders 
Internal  evaluation  and  accountability 

Outsiders  appear  to  be  motivated  to  evaluate  mainly  by  the 
desire  to  achieve  accountability  and  ultimately  responsibility  to 
concerns  such  as  whether  they  are  getting  their  money's  worth 
of  services.  By  way  of  contrast,  questionnaire  data  was  presented 
in  the  last  chapter  which  showed  that  neither  accountability  to 
community  nor  accountability  to  clientele  is  ranked  as  being  the  most 
important  function  of  evaluation  by  organizational  members.  Internal 
uses  of  evaluation,  such  as  evaluation  for  the  purposes  of  feedback 

and  program  evaluation,  are  preferred  by  people  within  the  organiza- 

..  22 
tion. 

Additional  questions  were  asked  related  to  accountability.  The 


21 

Of  course,  many  community  and  district  board  members 
strongly  support  funding  for  the  programs  of  the  mental  health 
centers.  The  point  is  that  they  are  not  limited  by  the  survival 
concerns  of  the  organization. 

22 

As  noted  previously,  these  functions  of  evaluation  are  not 
necessarily  mutually  exclusive.  Program  evaluation  simply 
refers  to  evaluation  in  which  the  program  is  the  unit  of  analysis. 
However,  the  preference  of  internal  organizational  members 
for  feedback  and  program  evaluation  over  accountability  reveals 
a difference  in  the  emphasis  of  the  evaluation. 
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responses  show  a strong  support  for  the  use  of  internal  evaluation 
to  achieve  accountability  to  the  community  and  clientele  of  the  mental 
health  organization.  Tables  5.  4 and  5.  5 show  that  over  80  and  70 
percent  of  clinicians  respectively  support  the  use  of  internal 
evaluation  to  determine  the  funding  of  the  mental  health  organizations 
and  to  achieve  a greater  degree  of  community  control,  support 
drops  to  57  and  47  percent  respectively  among  clinicians  as  revealed 
in  Tables  5.  6 and  5.  7.  It  should  be  noted  that  Table  5.  7 shows  that 
more  directors  oppose  the  use  of  evaluation  for  increased  community 
control  than  support  it.  This  is  significant  in  view  of  the  fact  that 
they  are  the  ones  who  usually  report  directly  to  the  district  boards. 
As  was  pointed  out  in  Chapter  III,  the  personnel  of  the  organizations 
show  strong  support  for  broad  evaluations  but  are  less  enthusiastic 
when  the  potential  threats  resulting  from  an  evaluation  are  evident. 

The  above  data  suggest  that  evaluators,  clinicians,  and  directors 
do  not  regard  evaluation  for  the  purpose  of  accountability  to  be  the 
most  important  reason  for  undertaking  evaluation  but  prefer  internal 
uses  of  evaluation.  A basic  factor  which  limits  the  kinds  of  informa- 
tion with  which  they  are  willing  to  provide  the  outside  community  is 
the  need  to  protect  the  organization  from  danger  (e.g.  , decline  in 
funds,  a bad  press,  and  so  forth).  Directors  and  other  personnel 
of  the  organizations  feel  it  is  important  to  present  only  positive 
information  concerning  their  organization  to  the  community: 
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TABLE  5.  4 

ACCOUNTABILITY  TO  COMMUNITY 


Evaluation  can  and  should  be  used  to  provide  accountability  of 
the  mental  health  organization  to  the  community  it  serves. 

Response 

Percentage  of 
Clinicians* 

Percentage  of 
Directors 

Percentage  of 
Evaluators* 

Strongly  Agree 

15.  8 (6) 

37.  5 (6) 

44.  4 (4) 

Agree 

71.  1 (27) 

37.  5 (6) 

55. 5 (5) 

Undecided 

5.  3 (2) 

12. 5 (2) 

— 

Disagree 

7.  9 (3) 

12. 5 (2) 

— 

Strongly  Disagree 

— 

— 

— 

*Sum  of  percentage  figures  is  unequal  to  100  percent  because  of 
rounding. 


TABLE  5.  5 

ACCOUNTABILITY  TO  CLIENTELE 


Evaluation  can  and  should  be  used  to  provide  the  accountability 
of  the  clinical  staff  to  the  clientele  of  the  mental  health  organization. 


Response 

Percentage  of 
Clinicians* 

Percentage  of 
Directors* 

Percentage  of 
Evaluators* 

Strongly  Agree 

15.  8 (6) 

25.  0 (4) 

22. 2 (2) 

Agree 

57. 9 (22) 

50.  0 (8) 

55. 5 (5) 

Undecided 

15.  8 (6) 

6.  3 (1) 

11.  1 (1) 

Disagree 

5.  3 (2) 

12.  5 (2) 

11. 1 (1) 

Strongly  Disagree 

5.  3 (2) 

6.  3 (1) 

— 

*Sum  of  percentage  figures  is  unequal  to  100  percent  because  of 

rounding. 
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TABLE  5.  6 

EVALUATION  FOR  FUNDING 


Evaluation  can  and  should  be  used  to  determine  how  much  funding 
mental  health  organizations  should  receive. 


Response 

Percentage  of 
Clinicians 

Percentage  of 
Directors* 

Percentage 

Evaluators 

Strongly  Agree 

23. 7 (9) 

6.  3 (1) 

22. 2 (2) 

Agree 

34.  2 (13) 

31.  3 (5) 

44.  4 (4) 

Undecided 

23. 7 (9) 

31.  3 (5) 

11.  1 (1) 

Disagree 

18.  4 (7) 

31.  3 (5) 

22. 2 (2) 

Strongly  Disagree 

— 

— 

— 

*Sum  of  percentage  figures  is  unequal  to  100  percent  because  of 
rounding . 


TABLE  5.  7 

EVALUATION  FOR  COMMUNITY  CONTROL 


Community  control  over  mental  health  services  can  and  should 
be  increased  through  the  use  of  evaluation. 


Response 

Percentage  of 
Clinicians* 

Percentage  of 
Directors  * 

Percentage  of 
Evaluators* 

Strongly  Agree 

2.  6 (1) 

6.  3 (1) 

22.2  (2) 

Agree 

44.  7 (17) 

31. 3 (5) 

22. 2 (2) 

Undecided 

28.  9 (11) 

12.  5 (2) 

33.  3 (3) 

Disagree 

21. 1 (8) 

37.  5 (6) 

22. 2 (2) 

Strongly  Disagree 

2.  6 (1) 

12. 5 (2) 

— 

*Sum  of  percentage  figures  is  unequal  to  100  percent  because  of 
rounding. 
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(Director)  I rather  doubt  that  most  people  in  the  community 
can  use  data  effectively  and  I imagine  that  accountability  to 
the  community  will  still  have  some  flowery  frills  involved,  and 
there  will  not  be  much  emphasis  placed  on  negative  components 
of  an  evaluative  study.  It  would  apparently  be  un-American 
to  do  so.  You  need  salesmanship- -selling  your  strong  points 
and  not  talking  about  your  weak  points.  That  is  American  culture, 
you  know.  It  is  actually  a sign  of  strength  to  display  your 
weaknesses  but  that  would  be  un-American.  Certainly,  it  is 
going  to  be  slanted  any  time  you  need  public  relations. 

Thus,  the  kinds  of  information  which  these  organizations  are  willing 

to  provide  voluntarily  to  the  community  or  other  outsiders  is  limited 

by  their  perception  of  the  need  to  present  a "good  face"  to  the  funding 

agencies  and  district  boards. 

Does  this  mean  that  the  centers  are  unresponsive  to  the  values 
of  the  community?  Are  the  organizations  less  critical  in  their 
evaluations  of  the  services  they  provide  than  the  outside  agencies 
and  boards  are?  Most  of  the  organizations  are  undertaking  formal 
evaluation  which  opens  them  up  to  feedback  and  evaluation  by  their 
clientele  through  the  use  of  goal  attainment  scaling,  consumer 
satisfaction  surveys,  and  needs  assessment  surveys.  Of  course, 
as  pointed  out  earlier,  these  evaluations  are  expected  to  be  positive 
and  are  carried  out  to  a great  extent  with  a view  to  supporting  the 
survival  of  the  organization. 

Nevertheless,  insofar  as  the  staff  took  the  information  derived 
from  these  evaluations  into  account  in  the  delivery  of  their  services, 
these  internal  evaluations  would  have  the  effect  of  making  the  personnel 
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of  the  organizations  responsive  (to  a certain  extent)  to  the  values  of 

23 

some  members  of  the  community- -the  clientele.  Also,  the  reason 
why  the  evaluations  are  likely  to  be  positive  is  due  more  to  their 
nature  than  to  any  overt  manipulation  by  the  organizational  personnel. 

Professionalism  and  evaluation 

While  it  was  argued  in  the  last  chapter  that  much  of  the  basis 
of  interest  in  formal  evaluation  is  related  to  the  concern  for 
organizational  survival,  there  are  other  sources  of  motivation  which 
can  potentially  foster  the  desire  to  evaluate  the  activities  and  services 
carried  out  by  the  mental  health  organizations.  The  issue  of  pro- 
fessionalism is  relevant  here  because  these  organizations  are  com- 
posed substantially  of  professionals  such  as  clinical  psychologists, 
psychiatrists,  and  psychiatric  social  workers,  among  others. 

Two  lines  of  argument  have  been  developed  concerning  the 
relationship  between  professionalism  and  evaluation.  On  the  one 
hand,  professionalism  has  recently  been  seen  as  an  obstacle  to 
evaluation  because  neither  professional  self-evaluation  nor  evaluation 
by  professional  colleagues  has  been  effective  in  controlling  the  quality 
of  professional  services.  ^ On  the  other  hand,  the  traditional  argument 

23 

"Community"  is  an  extremely  difficult  term  to  define.  The 
directors  tended  to  associate  the  "community"  with  the  district  boards. 

^See  Elliot  Freidson,  Professional  Dominance  (New  York: 
Atherton,  1970). 
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is  that  professionals  do  internalize  values  concerning  correct 

practices  and  quality  of  service  during  their  education  and  training 

25 

which  they  carry  over  into  their  practice. 

In  regard  to  the  first  point  of  view,  data  from  questionnaires 
was  presented  in  Chapter  III  which  shows  that  clinicians  and  directors 
perceive  professionalism  to  be  the  least  important  of  five  obstacles 
to  formalized  evaluation.  Nor  did  the  evaluators  see  professionalism 
as  being  a particularly  important  obstacle.  They  rate  the  lack  of 
resources  for  evaluation,  skepticism  concerning  the  technology  of 
evaluation,  and  anxiety  concerning  the  way  that  evaluation  may  be 
used  as  being  more  important  obstacles  to  the  effective  implementa- 

p O 

tion  of  an  evaluation  program. 

Two  additional  questions  were  asked  related  to  the  issue  of 
professionalism  and  formal  evaluation.  Table  5.  8 shows  that 
clinicians,  directors,  and  evaluators  reject  the  position  that  staff 
members  are  capable  of  evaluating  their  own  performance.  However, 
Table  5.  9 shows  more  uncertainty  in  regard  to  this  point. 


^Herbert  Simon  argues  that  the  professionalization  of  the 
federal  bureaucracy  has  led  to  increased  quality  of  service  and 
less  corruption.  See  Herbert  A.  Simon,  "The  Changing  Theory  and 
Changing  Practice  of  Public  Administration,  " in  Contemporary 
Political  Science;  Towards  Empirical  Theory,  ed.  Ithiel  de  Sola  Pool 
(New  York:  McGraw-Hill  Book  Company,  1967),  pp.  86-120. 

p c* 

It  should  be  noted  that  the  questionnaire  referred  specifically 
to  internal  evaluation  and  not  outsider  evaluation. 
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TABLE  5.  8 
PROFESSIONALISM 


Professional  clinical  staff  members  are  capable  of  evaluating 
their  own  performance  without  the  aid  of  a formal  evaluation  by  an 
evaluator. 


Response 

Percentage  of 
Clinicians* 

Percentage  of 
Directors* 

Percentage  of 
Evaluators* 

Strongly  Agree 

— 

— 

— 

Agree 

28.  9 (11) 

12. 5 (2) 

— 

Undecided 

23.  7 (9) 

25.  0 (4) 

11.  1 (1) 

Disagree 

44.  7 (17) 

43.  8 (7) 

55.  5 (5) 

Strongly  Disagree 

2.  6 (1) 

18.  8 (3) 

33. 3 (3) 

*Sum  of  percentage  figures  is  unequal  to  100  percent  because  of 
rounding. 

TABLE  5.  9 

CLINICAL  SELF-EVALUATION 

Only  the  clinical  staff  involved  in  direct  services  can  actually 
judge  whether  or  not  programs  are  successful. 

Response 

Percentage  of 
Clinicians 

Percentage  of 
Directors* 

Percentage  of 
Evaluators  * 

Strongly  Agree 

5.  3 (2) 

6.  3 (1) 

— 

Agree 

31.  6 (12) 

18.  8 (3) 

33. 3 (3) 

Undecided 

36.  8 (14) 

50. 0 (8) 

44.  4 (4) 

Dis  agree 

26.  3 (10) 

25.  0 (4) 

11. 1 (1) 

Strongly  Disagree 

— 

— 

11. 1 (1) 

*Sum  of  percentage  figures  is  unequal  to  100  percent  because  of 
rounding. 
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Interviews  and  participant  observation  indicate  that  many  of  the 
professionals  working  in  these  organizations  do  believe  in  what  one 
called  the  "professional  assumption"  that  they  can  manage  themselves. 
For  these  personnel,  the  acceptance  of  formal  evaluation  by  inside 
or  outside  evaluators  is  based  more  upon  practical  necessity  than 
anything  else.  The  following  director  and  supervisory  clinician 
took  this  point  of  view: 

(Supervisory  Clinician)  It  is  probably  true  [the  argument 
about  professionals  being  able  to  evaluate  themselves]  but  I 
don't  think  that  there  is  any  way  that  you  can  sell  that  to  the 
community.  They  would  say,  "These  are  queer  birds.  They 
don't  have  any  way  of  measuring  what  they  are  doing.  " 

(Director)  Sure,  it  is  prevalent.  Most  professionals  feel 
that  as  a component  of  their  professional  abilities  or  stature 
they  should  be  given  what  is  called  a professional  assumption 
that  they  will  manage  themselves  within  a professional  code 
of  ethics,  that  they  will  use  their  time  reasonably  efficiently 
and  competently.  . . . The  problem  is  that  if  we  are  going  to 
continue  to  receive  public  funds,  we  have  to  make  some  kinds 
of  attempts  to  evaluate  ourselves. 

There  does  not  appear,  then,  to  be  strong  resistance  to  the  idea 
of  formal  evaluation  by  inside  or  outside  evaluators  on  the  basis 
of  the  argument  that  such  evaluation  is  unnecessary  because 
professionals  can  and  do  evaluate  themselves.  However,  it  is  not 
clear  to  what  extent  this  acceptance  of  formal  evaluation  is  based 
upon  a belief  for  such  evaluation  or  the  pragmatic  acceptance  of 
formal  evaluation  because  it  is  necessary  to  obtain  public  funds. 


Critical  internal  evaluation 


The  above  discussion  leaves  unanswered  the  question  of  whether 
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professional  training  leads  these  professionals  to  actually  carry  out 
critical  self-evaluations  or  evaluations  of  other  programs  and  personnel 
within  the  organizations.  Also,  are  there  any  other  sources  of 
motivation  for  critical  evaluation  within  the  organization? 

Close  observation  of  one  Center  and  interviews  with  personnel 
at  other  organizations  brought  to  light  several  examples  of  clinicians 
making  highly  critical  judgments  about  the  quality  of  personnel  and 
programs  within  the  organizations  of  which  they  are  members.  How- 
ever, these  judgments  are  generally  of  informal  and  subjective  types 

O (-7 

of  "evaluation''  similar  to  those  described  in  Chapter  III.  ' For 
example,  a program  head  admitted  to  other  staff  members  of  the 
Center  that  the  program  he  headed  was  of  doubtful  quality  and 
represented  "band  aid"  treatment.  In  another  case,  a director  of 
a program  stated  to  other  program  heads  that  his  own  staff  had  had  only 
one  productive  day  since  he  had  taken  charge  of  the  program.  At 
another  Center,  the  director  describes  how  he  is  attempting  to  get 
his  own  answers  to  questions  related  to  effectiveness  aside  from  the 
information  required  by  the  State: 

(Director)We  want  to  get  the  answers  to  these  questions. 

I think  that  we  are  hungry  for  them  which  is  the  reason  why 
we  are  going  to  spend  what  to  me  is  a lot  of  money  on  this 
computer  firm  just  so  we  will  basically  know  what  is  going  on, 
what  kind  of  dropout  rate  we  have,  what  is  our  success  in 
various  programs.  . . 


27 

In  Chapter  III,  it  was  found  that  the  most  common  form  of 
evaluation  is  informal  and  subjective.  Here,  the  motivations  behind 
this  form  of  evaluation  are  analyzed. 
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Certain  types  of  informal,  critical  judgments  tend  to  be  more 
common  than  others  in  the  organization  where  participant  observa- 
tion was  carried  out.  In  particular,  the  supervisory  staff  spend  a 
great  deal  of  time  discussing  the  quality  of  the  performance  of  those 
lower  in  the  hierarchy  than  themselves.  They  discuss  how  to 
evaluate  nonprofessional  staff  in  the  organization  and  provide  them 
with  the  incentive  to  perform  better.  The  director  of  the  organization 
remarks  on  this  tendency  to  focus  evaluative  judgments  on  lower 

level  personnel  in  the  organization  but  his  point  is  not  taken  up  by 
2 8 

others  present: 

(Supervisory  Clinician  #1)  In  line  with  evaluation,  we  are 
trying  to  take  a look  at  each  of  our  clients  and  see  what  we 
can  do.  . . . What  is  their  potential? 

(Director)  How  go  we  evaluate  ourselves?  It  is  comforting 
to  talk  about  the  paraprofessionals  on  the  staff.  . . . 

(Supervisory  Clinician  #2)  I would  like  to  deal  with  the  issue 
of  consolidation. 

Another  type  of  highly  critical,  informal  evaluation  involves 
negative  statements  made  by  the  personnel  of  one  program  concern  ing 
the  quality  and  effort  of  other  programs.  An  example  occurs  in  the 
following  discussion  as  to  whether  locally  based  programs  should 
provide  their  own  "replica"  of  services  heretofore  provided  only 
at  the  central  location  of  the  organization. 


28of  course, lower  level  personnel  also  evaluate  their  superiors 
but  their  judgments  tend  to  focus  on  factors  such  as  the  degree  of 
protection  which  the  head  provides  to  the  organization  and  members  of 
their  own  program.  See  Herbert  A.  Simon,  Donald  W.  Smithburg,  and 

Victor  A.  Thompson,  Public  Administration  (New  York:  Alfred  A. 
Knopf,  1950),  p.  409. 
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(Supervisory  Clinician  #1)  We  don’t  look  at  the  problem  of 
quality.  Is  there  a feeling  that  we  can't  cut  out  any  programs? 

(Supervisory  Clinician  # 2 ) I think  that  there  is  a lot  of 
competition  going  on  and  they  are  trying  to  deliver  services 
but  can’t  furnish  the  quality. 

(Supervisory  Clinician  #3)  The  question  is  how  much 
community  based  service  we  are  going  to  have. 

In  another  example,  the  head  of  programs  in  a certain  area 

is  arguing  that  he  and  his  staff  involved  in  the  direct  delivery  of 

service  carry  a heavier  load  than  the  administrative  staff  (including 

the  evaluators)  and  consultants  in  the  organization: 

(Supervisory  Clinician)  I do  the  work  on  getting  local  grants 
funded  on  my  own. 

(Evaluator)  My  staff  would  do  this  if  you  made  it  a priority. 

(Supervisory  Clinician)  You  could  assist  me  but  I would 
have  to  put  in  so  much  input  myself,  that  when  it  came  down  to 
it,  it  would  fall  on  me. 

(Director)  There  seems  to  be  an  atmosphere  that,  "I  am 
working  hard  but  somewhere  else  someone  is  goofing  off.  " 

This  seems  to  be  prevalent. 

Still  another  source  of  motivation  for  informal,  critical 
evaluation  is  the  situation  in  which  personnel  within  the  organization 
disagree  with  the  organizational  "philosophy  of  treatment"  or  even 
have  doubts  about  the  worth  of  its  services.  Some  directors  admit 
to  feeling  that  private  practitioners  could  provide  better  quality  and 
more  quantity  of  direct  service  for  a similar  cost.  The  following 
director  takes  this  point  of  view: 
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(Director)  Well,  I will  give  you  an  example.  In.  . . region 
[one  of  the  areas  served  by  the  Center],  they  have  a budget 
of  x dollars.  A clinician  here  should  see  at  least  twenty-five 
hours  of  direct  therapy,  but  a professional  in  private  practice 
sees  patients  for  about  thirty-four  hours.  ...  I am  not  talking 
about  indirect  services.  . . but  you  are  paying  [figure  concerning 
the  number  of  dollars  spent  by  the  organization  in  this  region] 
and  it  seems  to  me  that  a psychiatrist  could  serve  that  population 
as  well  or  maybe  better.  29 

The  high  rate  of  turnover  in  the  position  of  director  (often  held  by 
psychiatrists)  may  have  been  due  in  part  to  the  rejection  by  directors, 
and  particularly  psychiatrists,  of  the  values  and  worth  of  the  mental 
health  center  movement.  ou  Disagreement  with  the  values  and  worth 
of  an  organization  makes  possible  a much  more  critical  perspective 
than  normally  would  be  expected  of  an  organizational  member. 

A final  source  of  motivation  to  evaluate  critically  the  services 
of  the  organization  is  ambition  on  the  part  of  members  of  the  organiza- 
tion, particularly  the  evaluators.  The  following  evaluator  admits 
to  seeing  his  major  avenue  of  career  development  to  be  at  the  state 
level  rather  than  within  the  organization.  He  welcomes  the  possibility 
of  the  state  agency  expanding  its  evaluation  of  local  mental  health 
centers: 


2 Q 

It  appeared  that  the  same  directors  who  were  skeptical  of  the 
quality  of  the  services  of  the  mental  health  centers  also  tended  to  be 
doubtful  of  the  worth  of  indirect  services. 

. 30 

Many  directors  feel  that  they  have  lower  status  after  leaving 
private  practice  to  become  heads  of  these  organizations.  See  H.  G. 
Whittington,  "Community  Mental  Health  Services  in  Kansas:  Personal 
Value  Systems  and  Mental  Health,  " American  Journal  of  Public  Health, 
Vol.  55  (January,  1965),  pp.  33-37. 
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(Evaluator)  Praise  the  Lord.  I think  it  is  definitely 
possible  [ i.  e.  , setting  up  an  expanded  state  evaluation  system] 
and  I hope  I can  be  part  of  it.  . . . I think  that  if  they  [ the 
state  agency]  bring  in  a group  of  evaluators  or  people  who  have 
had  experience  in  evaluation,  a group  like  this  would  probably 
come  up  with  some  reasonable  methodology.  ...  It  would 
require  an  increased  audit  by  the  State  and  a common  records 
system. 

In  short,  individuals  who  nominally  are  members  of  the  organ  ization 
can  remain  quite  detached  from  it  due  to  personal  ambitions  or 
disagreement  with  the  organization's  philosophy. 


Summary  of  Motivation 

An  evaluator  must  be  motivated  in  order  to  achieve  responsiveness 
to  the  values  and  priorities  which  he  desires  to  emphasize.  Out- 
siders and  insiders  tend  to  have  different  kinds  of  motivation  behind 
their  evaluations.  Outsiders  do  not  have  to  worry  about  the  survival 
of  the  organization.  They  tend  to  favor  what  is  called  a "summative 

evaluation"  in  which  the  purpose  is  to  render  a single  judgment  on 

31 

the  services  of  the  organization.  Their  major  constraints  are 
time  and  money  with  which  to  carry  out  the  evaluation. 

Inside  evaluators  are  usually  limited  by  attachment  to  organizational 

SI 

For  a discussion  of  the  difference  between  summative  and 
formative  evaluations,  see  Jacqueline  Kosecoff  and  Carol  Fitzgibbon, 
"Many  a Slip,"  Evaluation  Comment,  Vol.  4 (December,  1973), 
pp.  6-8.  The  latter  refers  to  evaluation  which  has  the  purpose  of 
improving  programs,  personnel,  and  organizations  rather  than  of 
rendering  final  judgment  on  them. 


207 


survival.  They  prefer  to  use  evaluation  for  internal  purposes, 
such  as  feedback,  to  improve  programs  and  services.  This  type  of 
analysis  is  called  "formative  evaluation.  " Thus,  organizations  do 
not  appear  willing  to  present  information  voluntarily  to  outside 
agencies  or  district  boards  which  would  reflect  unfavorably  on  the 
performance  of  the  organization  and  its  personnel.  Consequently, 
the  degree  to  which  internal  evaluation  provides  the  "functional 
equivalent"  of  outside  evaluation  depends  upon  the  extent  to  which 
there  is  a congruence  between  internal  and  external  value  systems. 

Despite  the  limits  imposed  by  concern  with  organizational 
survival,  critical  judgments  do  take  place  within  the  organizations 
under  study.  However,  they  are  generally  based  on  an  informal  and 
subjective  type  of  assessment.  The  basis  of  the  motivation  for 
critical  evaluation  includes  values  internalized  through  general 
education  and  professional  training.  Other  sources  of  motivation 
for  critical  evaluation  include  personal  ambition  and  disagreement 
with  the  values  of  the  organization.  Organizational  members  are 
also  generally  more  critical  of  persons  who  are  either  members  of 
different  programs  or  lower  in  the  hierarchy  than  themselves. 

Authority 

It  has  already  been  described  how  it  is  necessary  for  an 
evaluator  to  be  able  to  identify  and  measure  his  goals  in  order  to 
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achieve  responsiveness  to  them.  He  must  also  possess  the 

motivation  to  gain  responsiveness  to  his  values.  A third  factor 

which  is  important  in  determining  the  amount  of  responsiveness 

secured  by  the  evaluator  is  the  amount  and  kinds  of  authority  which 

he  exercises.  Authority  may  be  defined  as  the  "ability  to  evoke 

compliance  in  others.  " Broadly  defined,  there  are  several 

potential  bases  of  authority,  including  authority  of  position,  authority 

33 

of  competence,  and  authority  of  person. 

What  are  the  resources  of  an  evaluator  that  could  make  the 

rest  of  the  staff  responsive  to  him?  Based  on  a search  of  the 

literature  on  evaluation  and  information  from  participant  observation 

and  interviews,  five  factors  were  selected  which  are  potentially 

34 

important  in  determining  the  effectiveness  of  the  evaluator. 


3^See  Robert  Presthus,  "Authority  in  Organizations,  " Public 
Administration  Review,  Vol.  20  (Spring,  1960),  pp.  86-91. 

^See  Robert  L.  Peabody,  Organizational  Authority  (New  York: 
Atherton,  1964).  Authority  of  competence  can  include  both  competence 
learned  through  formal  education  and  competence  acquired  by  ex- 
perience. Authority  of  person  can  include  relationships  with  different 
groups  such  as  clinicians  and  directors  of  the  organizations. 

0 4 

Relating  these  five  factors  used  in  the  questionnaire  to  the 
three  types  of  authority  derived  from  Peabody,  "the  respect  of  the 
clinical  staff  for  the  evaluator's  technical  skills  and  knowledge" 
and  "formal  role  and  powers"  are  the  equivalent  of  authority  of 
competence  and  authority  of  position.  The  evaluators'  relationships 
with  the  clinical  staff  and  director  of  the  organization  may  both 
be  considered  subtypes  of  "authority  of  the  person.  " 
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Clinicians,  evaluators,  and  directors  were  asked  to  rank  the 

importance  of  these  five  factors  and  a rough  index  of  their  importance 

was  formed  as  described  in  Table  5. 10.  There  is  a general  consensus 

as  to  the  order  of  importance  of  the  different  functions  among  all 

three  groups  of  respondents.  The  evaluator’s  technical  competence 

and  support  from  the  head  of  the  organization  are  clearly  regarded 

35 

as  the  two  most  important  of  the  five  factors  listed. 

The  evaluator's  contribution  to  "non-evaluative"  activities 
is  regarded  as  being  least  important  of  the  five  factors.  The 
evaluator's  formal  powers  and  friendships  with  the  clinical  staff 
are  regarded  as  being  somewhat  less  important  than  the  respect 
of  the  clinical  staff  for  his  technical  competence  and  relationship 
to  the  director. 

Authority  of  Competence 

As  noted  earlier,  the  technology  of  evaluation  is  seen  as  needing 
further  development  but  is  not  generally  considered  to  be  an 
insuperable  obstacle  to  successful  evaluation.  Interviews  and 
participant  observation  confirm  that  the  perception  of  the  evaluator's 

3 5 

It  should  be  noted  that  the  questionnaire  refers  to  the  "respect" 
of  the  clinical  staff  for  the  evaluator's  competence  rather  than 
"objective"  competence. 
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TABLE  5. 10 

THE  RESOURCES  OF  AN  EVALUATOR 


Please  rank  the  following  in  what  you  regard  to  be  their  order 
of  importance  in  enabling  an  evaluator  to  establish  an  effective 
evaluation  program.  (1  = the  most  important;  5 = the  least 
important). 

#1  The  evaluator's  formal  role  and  powers  in  the  organization. 

#2  The  evaluator's  personal  friendships  and  relationships 
with  the  clinical  staff. 

#3  The  respect  of  the  clinical  staff  for  the  evaluator's 
technical  skills  and  knowledge. 

#4  The  evaluator's  relationships  with  and  support  from  the 
head  of  the  mental  health  organization. 

#5  The  evaluator's  contributions  to  "non -evaluative"  aspects 
of  his  job  such  as  data  collection,  grant  writing,  and 
administration. 

The  following  index  was  formed  by  scoring  five  points  for  each 
time  that  a factor  was  ranked  as  being  the  most  important,  four 
points  for  the  next  most  important,  etc. 


Clinicians  Directors  Evaluators 


Number 

Points 

Number 

Points 

Number 

Points 

3 

152 

3 

63 

3 

37 

4 

111 

4 

56 

4 

30 

1 

101 

2 

36 

1 

27 

2 

101 

1 

36 

2 

24 

5 

80 

5 

34 

5 

16 
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competence  by  the  clinical  staff  and  director  is  a necessary  pre- 
condition for  his  effectiveness.  One  evaluator  admits  that  he  wishes 
he  had  a "bigger  bag  of  tricks,  " and  that  his  biggest  flaw  is  a lack 
of  time  to  explore  the  literature  of  evaluation: 

(Evaluator)  The  biggest  flaw  that  I see  in  what  I am  able 
to  do  is  that  I have  not  yet  had  the  opportunity  to  completely 
explore  the  literature,  that  I have  not  had  the  opportunity  to 
develop  to  my  satisfaction  a sufficient  inventory  of  methods.  . . 
and  that  is  my  concern--that  I have  a very  broad  knowledge  and 
a very  good  bag  of  tricks. 

However,  evaluators  tend  to  be  skeptical  of  the  ability  of  most 
of  the  clinical  staff  to  judge  their  knowledge  and  abilities  in  the 
evaluation  field.  The  following  evaluator  expresses  himself  on 
what  he  considers  to  be  their  lack  of  sophistication  concerning 
statistical  techniques : 

(Evaluator)  Well,  the  problem  is  that  a lot  of  clinicians -- 
and  also  the  non-clinical  people  who  hire  you- -don't  really 
understand  a sophisticated  data  analysis.  So,  if  you  give  them 
a report  that  is  in  terms  of  the  number  of  people  handled  and 
percentages  of  clients  who  say,  "Gee,  that  was  really  peachy, 
that  is  something  that  is  readily  intelligible  to  them.  And  they 
can  go  to  their  board.  . . and  say,  "Look,  isn't  this  nice? 
Everyone  thinks  we  are  so  wonderful.  We  handled  so  many 
clients.  ..."  I bet  that  when  I start  feeding  variables  in  and 
doing  something  like  a discriminant  analysis  that  there  are  not 
more  than  one  or  two  people  over  there  who  would  know  about 
that.  . . .36 


33A  discriminant  analysis  is  a multivariate  technique  in  which 
the  independent  variables  are  used  to  predict  which  of  two  or  more 
mutually  exclusive  categories  the  cases  will  "fit  into,  " See  Francis 
J.  Kelley,  Donald  Beggs,  and  Keith  A.  McNeil,  Research  Design  in 
the  Behavioral  Sciences:  Multiple  Regression  Approach  (Carbondale 
and  Edwards ville:  Southern  Illinois  University  Press,  1969), 
pp.  234-240. 
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In  addition  to  having  evaluators  with  competent  technical  ability 
and  knowledge,  clinicians  also  favor  persons  with  clinical  knowledge 
and  experience.  In  one  organization,  the  director  was  discussing 
the  possibility  of  giving  the  evaluator  some  "line  authority"  and 
thus  ability  to  make  substantive  decisions  concerning  the  programs. 
Some  clinicians  objected  because,  they  said,  the  evaluator  had  no 
"clinical  experience"  or  "clinical  appreciation": 

(Supervisory  Clinician)  What  are  you  talking  about?  [con- 
cerning the  evaluator's  proposed  new  responsibilities],.  . . 
putting  him  in  a staff  or  line  position? 

(Director)  For  example,  I would  go  out  and  work  in  the 
clinical  area  and  talk  about  programs,  not  about  dollars  and 
cents,  planning,  or  effectiveness. 

(Supervisory  Clinician)  I am  opposed  to  that  and  part  of 
the  problem  is  that  [name  of  the  evaluator]  has  had  no  clinical 
experience.  ...  He  has  no  clinical  appreciation. 

Given  the  choice,  all  clinicians  interviewed  agree  that  they 

would  prefer  someone  with  clinical  experience  as  an  evaluator. 

Several  point  out  that  such  individuals  are  hard  to  find: 

(Supervisory  Clinician)  But  you  really  need  somebody  who 
knows  research  design,  and  most  good  clinicians  that  I know 
don't  know  or  don't  like  research  and  development.  I don't 
know  why  but  they  seem  to  be  mutually  exclusive. 

One  evaluator  (without  clinical  experience  or  background)  sees  the 

argument  concerning  clinical  experience  as  merely  a legitimating 

factor  in  overcoming  resistance  to  the  conclusions  of  the  evaluator: 
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(Evaluator)  Clinical  skills  do  help  and  clinical  credentials 
do  help.  It  is  because  you  become  more  legitimate,  and  it  is 
very  easy  for  people  to  point  to  you  and  say,  "Well,  we  don't 
have  to  listen  to  him  because  he  isn't  a clinician.  ..." 

The  perception  of  competence  and  knowledge  can  be  more 

important  than  their  actual  possession.  An  evaluator  who  had  a 

doctoral  degree  in  psychology  said  that  the  clinicians  in  the 

organization  assume  that  he  has  knowledge  concerning  the  clinical 

field  because  his  degree  is  in  psychology.  However,  he  actually 

did  not  have  this  experience.  37  Earlier,  it  was  pointed  out  that 

program  directors  and  personnel  prefer  to  have  an  evaluator  who 

comes  and  "lives  with  their  program"  and  thus  sees  its  problems 

and  limitations.  Generally,  it  appears  that  clinicians  prefer  to  have 

someone  evaluating  them  who  shares  as  closely  as  possible  their 

own  background,  knowledge,  and  general  perspective. 

Authority  of  Position 

Responses  to  the  questionnaire  indicate  that  formal  authority 
associated  with  the  position  of  the  evaluator  is  not  nearly  as  important 
as  the  evaluator's  relationship  with  the  head  of  the  mental  health 
organization.  This  finding  is  not  surprising  because,  as  was  pointed 
out  in  Chapter  IV,  it  is  the  director  of  the  mental  health  center  who 

T7 

The  disciplines  and  backgrounds  of  the  evaluators  were 
varied,  including  clinical  psychology,  education,  systems  analysis, 
sociology,  and  social  work. 
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largely  determines  whether  to  invest  in  an  evaluator  and  formal 
evaluation. 

The  actual  structure  of  these  organizations  varies  considerably 
despite  the  fact  fact  that  they  offer  the  same  basic  core  of  services. 

In  many  organizations  the  evaluator  and  his  staff  are  regarded  as 
nominally  "co-equal,  " in  terms  of  formal  authority,  with  the  program 
heads  and  staff  of  other  programs.  However,  the  amount  of  resources 
devoted  to  evaluation  is  usually  relatively  small  in  comparison  with 
the  other  service  programs.  Thus  most  evaluators  did  not  perceive 
their  status  to  be  the  equivalent  of  that  of  the  other  program  heads. 

The  importance  attached  to  the  evaluator's  particular  relation- 
ship with  the  head  of  the  organization  is  especially  significant 

3 8 

because  there  is  a large  turnover  in  the  latter  position.  A 
change  in  the  director  of  the  organization  can  in  large  part  determine 
whether  an  evaluator  will  be  effective  in  his  role. 

Authority  of  the  Person 

In  our  above  discussion  concerning  positional  authority, 
it  was  noted  the  evaluator  must  develop  a good  relationship  with  the 
director  in  order  to  be  effective.  The  two  other  factors  associated 

3 8 

During  the  first  six  months  of  this  study,  four  of  the  director- 
ships of  the  fourteen  organizations  under  study  either  were  already 
vacant  or  became  vacant. 
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with  authority  of  the  person--the  evaluator's  personal  relationships 

with  the  clinical  staff  and  his  contributions  to  non -evaluative 

39 

activities --are  less  important.  Nevertheless,  interviews  and 

participant  observation  indicate  that  substantial  importance  is 

attached  to  these  two  factors  by  both  evaluators  and  clinicians. 

Introducing  evaluation  by  beginning  with  non -threatening  types 

of  evaluation  and  securing  the  participation  of  the  clinical  staff 

are  two  of  the  major  strategies  used  by  evaluators:^® 

(Evaluator)  You  have  probably  heard  in  the  past  that  they 
experienced  problems  in  research  and  evaluation  and  couldn't 
get  the  staff  to  do  this  and  that.  The  thing  that  I think  that  has 
enabled  us  to  get  past  that  is  to  do  a study  that  is  in  no  way 
threatening  to  the  staff.  They  could  care  less.  When  they  saw 
that  I was  a fairly  decent  guy,  that  I wasn't  out  to  be  a threat 
to  them.  . . They  would  say,  "Maybe  he  has  got  something 
to  offer  us.  " 

The  director  of  the  organization  in  which  the  above  evaluator  operates 
agrees  with  this  method  of  introducing  evaluation: 

(Director)  It  wasn't  planned  this  way  but  I think  that 


O Q 

As  used  here,  "authority  of  person"  refers  to  differences 
among  individuals  in  their  ability  to  secure  compliance  from  others 
which  is  not  dependent  on  the  particular  position  which  the  individuals 
hold  or  on  their  technical  competence.  While  nominally  called 
authority  of  person,  it  depends  not  only  on  his  personality  but  also 
his  ability  to  develop  relationships  with  others. 

40 

Introducing  evaluation  has  often  created  problems  of  resistance. 
See,  e.g.  , Richard  H.  Blum  and  Joseph  J.  Downing,  "Staff  Response  to 
Innovation  in  a Mental  Health  Service,  American  Journal  of  Public 
Health,  Vol.  54  (August,  1964),  pp.  1230-1240. 
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bringing  in  a research  and  evaluation  staff  to  do  a needs 
assessment  study  first  is  a real  plus  because  we  are  not 
evaluating  anyone  but  the  community.  . . . If  he  had  come  on 
board  and  immediately  talked  to  the  staff  on  how  we  are  going 
to  evaluate  programs  he  might  have  received  a very  different 
reception.  . . . 

Personality  and  interpersonal  relationships  are  important 

factors  in  determining  the  effectiveness  of  role  performance  by 

41 

individuals  in  organizations.  Some  argue  that  these  two  factors 

42 

are  especially  important  in  organizations  composed  of  clinicians. 

Certainly,  the  evaluators  are  quick  to  admit  their  importance: 

(Evaluator  #1)  I would  argue  that  the  informal  networks 
are  much  more  important  in  a place  like  this  than  the  more 
formal.  This  is  probably  true  in  most  organizations  but  here 
it  is  even  more  so  because  of  the  types  of  disciplines  represented. 
The  military  operates  on  informal  relationships  but  breaks 
these  down  every  so  often  by  moving  people  around. 

(Evaluator  #2)  No  matter  how  talented  you  are,  no  matter 
how  much  money  you  get,  and  how  much  you  know  about  research-- 
if  the  people  don't  trust  you  enough  in  order  to  cooperate  with 
you,  then  you  can't  do  anything.  They  have  to  cooperate.  They 
have  to  want  what  you  have  to  offer.  We  operate  in  response 
to  ongoing  situations,  and  if  we  can't  get  them  to  go  along  with 
us  or  to  feel  a need  of  what  we  have  to  offer  then  we  are  no 
good  at  all.  . . . 

(Evaluator  #3)  Without  the  ability  to  develop  good  inter- 
personal relationships,  you  just  don't  pull  off  an  evaluation 
program.  This  is  essential--to  get  cooperation  on  their  part. 


See  Chris  Argyris,  "The  Individual  and  the  Organization: 
Some  Problems  of  Mutual  Adjustment,  " Administrative  Science 
Quarterly,  Vol.  2 (June,  1957),  pp.  1-24. 

42 

See  Leonard  P.  Ullmann,  Institution  and  Outcome  (Oxford: 
Pergamon  Press,  1967),  p.  125. 
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Thus  evaluators  see  it  as  essential  to  supplement  their  technical 

knowledge  and  relationship  with  the  director  with  good  interpersonal 

43 

relationships  with  the  clinical  staff. 

One  evaluator  described  himself  as  a "marginal  man"  in  the 
organization.  Generally,  both  evaluators  and  other  staff  see  an 
important  difference  between  clinicians  who  "produced"  (e.  g.  in 
the  form  of  therapy)  something  immediately  of  value  for  the  organiza- 
tion and  researchers  whose  product  and  worth  is  less  easily  defined 
or  measured.  An  awareness  concerning  this  point  of  difference  is 
plainly  evident  in  the  following  comments  of  a director  and  evaluator: 

(Director)  Research  takes  some  time  before  production 
comes  forth  and  this  isn't  true  of  clinicians.  When  we  hire 
one  usually  the  first  day  or  two  they  have  seen  a couple  of 
patients  and  have  produced  something. 

(Evaluator)  I am  a member  of  the  planning  staff  [of  the 
entire  organization]  and  as  director  of  research  and  evaluation 
I am  supposed  to  be  there  at  the  meeting.  I want  to  be  there 
but  I won't  go  until  I have  something  to  take  with  me.  I look 
at  [name  of  clinician]  who  is  outpatient  director  and  he  provides 
a service,  sits  down  and  talks,  writes  out  an  admission  form, 
a progress  report,  and  he  has  done  something.  ...  I have 
been  attempting  to  develop  a credibility,  doing  small  kinds  of 
projects  for  the  outpatient  people.  . . . 

Another  evaluator  (at  a different  organization)  concurs  with  the  need 


43As  was  discussed  in  Chapter  III,  there  was  no  example  of 
the  network  of  informal  relations  directly  blocking  the  efforts  of 
the  evaluator.  Yet  the  informal  system  had  a very  important  effect 
in  that  the  evaluator  attempted  to  anticipate  and  avoid  any  potential 
anxiety  and  resistance. 
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to  "produce  something": 

(Evaluator)  I think  my  initial  way  of  doing  something  is  to 
get  at  it  quick,  provide  something  to  somebody  fast  so  that  the 
people  could  see  that  we  could  produce  something  that  they  could 
grab  hold  of.  . . . I think  the  key  to  the  job  is  being  productive 
and  visible  fast. 

It  was  suggested  in  the  last  chapter  that  the  evaluator's  per- 
formance of  "non -evaluative"  duties  is  to  a certain  extent  the  result 
of  the  necessity  to  deal  with  survival  needs  first  and  also  the  tendency 
of  programmed  activities  to  drive  out  "non-programmed"  activities. 
However,  another  reason  is  that  performing  "service  functions" 
for  the  directors  and  clinicians  help  evaluators  to  establish  personal 
relationships  with  other  staff  members.  It  is  also  a means  by 
which  evaluators  can  provide  a "concrete"  service  quickly  in  order 
to  justify  the  resources  spent  on  their  salaries.  The  irony  of  the 
situation  is  that  evaluators  are  being  responsive  to  the  needs  of  the 
clinicians  rather  than  the  other  way  around. 

While  this  service  approach  is  likely  to  enable  the  evaluators 
to  establish  good  relationships  with  the  clinicians,  evaluators  tend 
to  get  "locked  into"  a role  of  doing  service  and  descriptive  types  of 
activities.  They  find  it  difficult  to  escape  from  these  types  of 
activities  and  to  carry  out  more  formal  and  analytical  kinds,  like 
research  and  evaluation: 

(Evaluator)  What  happened  was  that  when  I started  doing  that, 
I got  myself  overloaded  with  a lot  of  descriptive  stuff,  and 
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didn't  get  into  any  actual  evaluation  for  several  months 
because  of  it.  As  a result,  a lot  of  people  kept  asking  me --they 
got  the  impression  that  that  [descriptive  activities]  is  what 
I am  supposed  to  do --so  they  kept  asking  me  to  do  all  of  this 
descriptive  stuff  such  as  compiling  the  ages  of  people  in 
programs.  Not  that  it  is  not  interesting  and  important  but  it 
doesn't  tell  you  all  that  much  about  how  your  programs  are 
doing.  ...  I was  using  that  initially  to  provide  them  public 
relations  and  a service  but  I ended  up  constantly  having  to  pro- 
vide them  a service. 

On  the  other  hand,  an  evaluator  who  uses  a more  "hard  sell" 
approach  received  a negative  reaction  from  the  clinical  staff  and 
the  director  of  the  organization: 

(Director)  [Name  of  the  evaluator]  does  nothing  to  court 

them  either.  . . . does  give  them  negative  feedback  but 

he  does  it  in  a way  that  makes  it  look  very  defensive  on  his 
part:  "I  couldn't  get  the  cooperation  of  the  therapist  here  and 

that  is  why  this  is  an  invalid  program,  " and  I say  to  , 

"What  did  you  do  to  get  the  cooperation?  " and  it  turns  out 
that  on  one  project,  the  therapist  didn't  even  know  that  he 
was  doing  the  project  and  yet  he  [the  evaluator]  labeled  him 
[therapist]  uncooperative. 

The  great  difficulty  of  performing  the  role  of  the  evaluator  is 
apparent  from  the  statements  above.  He  must  prove  his  worth 
to  the  organization  and  develop  good  relationships  with  clinicians. 

In  order  to  do  this,  he  has  to  compromise  his  role  from  that  of 
being  a "pure"  evaluator. 


The  Authority  of  External  Evaluators 
Authority  of  position 

The  nature  of  the  authority  exercised  by  the  outside  evaluators 
to  achieve  responsiveness  to  their  values  is  quite  different.  To 
achieve  their  demands,  they  relied  basically  on  their  formal  authority 
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in  the  form  of  control  over  funds.  These  outside  agencies  and  boards 

do  not  feel  it  necessary  to  develop  relationships  with  the  clinical 

staff  but  are  able  to  say,  in  effect,  "Carry  out  this  evaluation  if 

you  want  to  receive  money  from  us.  " 

The  centers  have  money  coming  from  several  sources  including 

44 

local,  state  and  national  governments.  Despite  these  multiple 
sources  of  funding  they  are  not  in  the  position  to  decline  a source 
of  funding.  Outsiders'  use  of  this  formal  authority  generated  some 
antagonism  and  resistance.  The  personnel  of  the  centers  viewed 
these  outside  requirements  for  accountability  as  being  mechanisms 
of  control  only.  They  spend  a great  deal  of  time  filling  out  forms 
from  which  they  do  not  receive  useful  feedback.  A program  director 
describes  his  reaction  to  one  example  of  data  collection  by  the  State: 

(Supervisory  Clinician)  We  were  told  by  those  in  charge 
of  dispensing  the  money  that  we  had  to  keep  a log  of  the  time 
we  spent  with  a certain  type  of  patient  for  a certain  month 
of  the  year.  We  wanted  our  money  and  so  we  went  after  it. 

Now  we  are  told  that  we  have  to  do  that  again.  Now  we  probably 
want  our  money  and  we  probably  will  go  after  it. 

At  a meeting  attended  by  state  officials,  a director  of  a local 

organization  told  these  officials  that  his  Center  was  going  to  evaluate 


The  fact  that  organizations  receive  multiple  sources  of 
funding  tends  to  make  the  job  of  internal  evaluators  more  difficult. 
See,  Lee  Yudin  and  Stephen  Ring,  "The  Impact  on  Research  and 
Evaluation,  " American  Journal  of  Orthopsychiatry,  Vol.  41  (January, 
1971),  pp.  149-157. 
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the  "cost  of  providing  the  information"  (which  was  required  by  the 
State  in  order  for  the  organization  to  receive  funding)  to  the  state 
agency.  Another  supervisory  clinician  saw  the  demands  made  by 
NIMH  for  evaluation  and  accountability  as  an  attempt  to  destroy 
the  mental  health  organizations: 

(Supervisory  Clinician)  The  worst  possible  source  for  the 
development  of  evaluation  is  Casper  Weinberger.  ^5  We  know 
Casper's  game  from  California.  He  sets  up  these  ponderous 
so-called  evaluation  techniques.  He  never  says  that  he  is 
going  to  cut  off  any  programs  but  he  just  says,  "Just  fill  out 
the  following  forms.  ..."  And  then  he  gets  you  going.  . . . 

In  short,  the  sole  reliance  on  formal  authority  by  outside  evaluators 

tends  to  provoke  reaction  and  resistance.  ^ 


Authority  of  competence 


There  is  strong  doubt  among  the  personnel  of  the  organizations 
that  the  local  communities  and  particularly  the  local  boards  possess 
the  knowledge  to  evaluate  the  quality  of  services  provided  by  the 
mental  health  centers: 

(Director  #1)  The  quality  of  service  has  not  been  looked 
at  by  the  community.  I can't  think  of  the  community  as  being 
quality-oriented.  They  talk  about  whether  or  not  services  are 
being  provided  but  not  necessarily  about  the  quality. 


45Casper  Weinberger  is  the  present  head  of  the  Department  of 
Health,  Education,  and  Welfare  (HEW). 

46 

Social  psychology  has  been  used  to  handle  reactions  against 
the  use  of  formal  authority  in  the  field  of  personnel  administration. 
See  Jacobo  A.  Varela,  Psychological  Solutions  to  Social  Problems 
(New  York:  Academic  Press,  1971),  p.  123. 
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(Director  #2)  The  district  board  would  have  no  capability 
of  carrying  out  an  evaluation  of  the  Center.  The  only  thing  I 
can  think  of  would  be  for  them  to  ask  the  Center:  "Well,  what 
are  you  doing?"  But,  for  them  to  evaluate  themselves  all  of 
the  programs,  I couldn't  imagine  them  doing  it. 

(Director  #3)  They  [the  district  board]  never  made  any 
criticism  of  programs  or  services.  Oh,  they  were  disturbed 
by  the  fact  that  emergency  services  weren't  being  provided,  but 
basically.  . . . There  has  been  nothing  generated  by  the  board 
of  a critical  nature  of  the  programs  and  services. 

The  boards  are  more  interested  in  financial  forms  of  account- 
ability and  responsibility  than  in  the  qualitative  aspects  of  services. 
The  personnel  of  the  organizations  usually  recognize  this  interest  in 
financial  accountability  to  be  a valid  and  important  area  for  the 
boards  to  "evaluate.  " In  the  area  of  monetary  concerns,  the  boards 
are  on  a more  equal  basis  of  understanding  with  the  center  personnel 
than  they  are  in  the  area  of  determining  the  quality  of  programs. 

Sometimes  the  district  boards  are  seen  as  becoming  overly 
focused  on  not  only  problems  of  money  but  "petty"  things  such  as 
catching  personnel  of  the  mental  health  organization  topping  during 
work  hours: 

(Psychiatrist)  Take,  for  example,  the  case  of  [name  of 
member  of  the  staff  of  the  mental  health  organization]. 

She  spends  ten  to  twelve  hours  a day  working  but  then  she  is 
caught  downtown  shopping  during  the  day  by  a member  of  the 
district  board. 

(Business  Manager)  We  are  always  on  the  defensive.  If 
they  think  they  can  do  better  with  someone  else.  . . . 

Some  of  the  questions  they  ask,  a kid  in  grade  school  wouldn't 
ask. 
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(Director)  They  use  the  budget  to  maintain  control. 

(Business  Manager)  We  want  to  be  members  of  what  is 
considered  to  be  a good  organization. 

(Director)  It  would  be  nice  if  the  board  would  focus  on 
programs.  . . . 

The  emphasis  by  the  district  boards  and  agencies  on  fiscal 

kinds  of  accountability  measures  led  one  director  to  declare  at  a 

meeting  attended  by  the  author  that  accountability  and  evaluation 

(qualitative)  were  completely  different:^ 

(Director)  I think  that  there  has  been  an  intermingling 
between  evaluation  and  accountability  in  this  discussion.  I 
don't  think  they  are  the  same  thing.  Let's  move  over  to  evalua- 
tion and  leave  accountability  to  the  fiscal  officers.  It's  really 
up  to  your  business  administrators.  I think  evaluation  and 
accountability  are  completely  different. 

Others  took  issue  with  this  director  by  arguing  that  there  are  more 

4ft 

kinds  of  accountability  than  simply  fiscal  accountability. 

The  state  and  national  agencies  are  seen  as  being  more 
knowledgeable  about  quality  of  service  than  the  local  boards.  Yet, 
these  agencies  simply  do  not  have  the  time  nor,  perhaps,  the  motivation 
to  spend  very  much  time  evaluating  one  organization.  Consequently, 


4^This  statement  occurred  at  a meeting  with  various  other 
directors,  evaluators,  and  officials  present.  Most  directors  did 
tend  to  associate  accountability  with  dollars  and  cents  issues  and 
also  quantitative  indices.  However,  in  a later  interview,  this  same 
director  admitted  that  there  were  other  forms  of  accountability 
than  simply  "fiscal  accountability.  " 

48 

The  plurality  of  interpretations  of  the  meaning  of  account- 
ability has  been  noted  by  others.  See,  e.  g.  , Kaplan  and  Roman,  The 
Organization  and  Delivery,  pp.  299-300. 
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they  tend  to  rely  on  two  major  forms  of  data  to  evaluate  these 
organizations:  (1)  selected  quantitative  criteria  such  as  cost  per 
encounter,  reduction  in  mental  hospital  loads,  and  so  forth,  and 
(2)  on-site  visits  by  representatives  from  the  agencies. 

Most  personnel  of  the  organizations  reject  the  idea  that  their 
organization  can  be  adequately  evaluated  by  selected  quantitative 
data: 

(Director)  I think  that  the  larger  the  unit  you  go  to,  the  more 
difficult  it  becomes  to  make  valid  comparisons.  I think  it  would 
be  difficult  enough  to  make  comparisons  within  a single  Center, 
but  when  you  extrapolate  out  statewide  and  try  to  use  a single 
system  to  make  comparisons  among  all  of  the  centers  and  clinics 
in  the  State  of  Florida,  it  may  be  conceivably  possible  but 
I would  hate  to  be  on  the  other  end.  . . . The  anxiety  is  always 
there  that  somebody  may  very  well  hold  it  up  and  say,  "That's 
it.  " That  is  the  risk  in  setting  up  any  kind  of  data  system-- 
that  some.  . . will  think  that  that  item  of  data  is  the  only  thing 
that  should  be  used  in  the  determination.  . . . 

'While  some  members  of  the  organization  feel  that  the  on-site 
visits  are  among  the  more  useful  forms  of  evaluation  because  of 
the  personal  contact  they  have  with  the  outside  evaluators,  others 
doubt  the  utility  of  this  form  of  evaluation  also: 

(Director  #1)  I am  even  more  skeptical  of  outside  evaluations. 
The  most  helpful  to  me  have  been  on-site  visits  by  NIMH  and 
the  State.  . . . 

(Supervisory  Clinician)  NIMH  comes  down  here.  . . . 

That  is  a disconcerting  thing  because  we  don't  have  the  time  to 
explain  to  them  and  so  forth.  It  is  funny.  You  don't  get  shaken 
by  it  but  you  know  that  they  are  not  understanding.  . . but  it 
never  seems  to  get  through  because  they  have  already  reached 
their  conclusions  about  what  they  think  you  should  be  doing  that 
you  are  not  doing.  They  don't  even  know  whether  you  can  do 
them  or  not.  Everyone  is  nice.  We  just  never  deal  with  one 
another.  We  are  just  nice. 
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(Director  #2)  When  NIMH  comes  down  once  or  twice  a 
year  to  consult,  you  get  a review,  but  it  is  a very,  very  gross 
one.  It  simply  sees  whether  or  not  you  are  doing  what  you  said 
you  were  going  to  be  doing  in  the  grant. 

(Director  #3)  So,  up  to  the  present  time,  the  NIMH  type  of 
visit  has  not  been  that  useful  to  us.  They  have  forced  us  to  do 
certain  things,  like  install  the  time-clocks,  which  the  pro- 
fessionals resent.  They  think  that  they  should  be  trusted. 

Without  exception,  all  clinicians  and  directors  interviewed 

prefer  inside  evaluations  to  those  done  by  outsiders.  In  large  part, 

this  is  undoubtedly  due  to  the  fact  that  they  perceive  the  former  to 

be  less  threatening.  Another  factor  is  that  inside  evaluations  are 

seen  as  being  more  capable  of  feeding  back  something  of  value  to 

the  organization  because  they  understand  its  complexity  and  decision 

making  system: 

(Supervisory  Clinician)  Results  can  be  read  within  terms  of 
their  total  knowledge.  Straight  numbers  often  don't  have  much 
meaning.  . . . And  we  just  have  more  confidence  in  somebody 
we  know  better  than  someone  else. 

(Director  #1)  If  you  want  to  do  any  in-depth  kind  of  thing, 
you  have  to  know  what  is  going  on.  Unless  you  want  to  send  in 
any  evaluation  team  to  stay  here  for  three  or  four  months  and 
work  out  an  evaluation --then  an  outsider  might  be  able  to  do  it. 

(Director  #2)  I am  not  interested  in  the  people  coming  in  for 
a one-shot  evaluation --that  kind  of  a thing  is  not  helpful.  Having 
a research  staff  on  a long  term  basis  could  assist--that  is  of 
interest  to  me. 

(Evaluator)  This  clinic  is  so  complex  that  it  takes  a very 
long  time  to  learn  about  it--which  you  really  should  do  to  set 
up  good  evaluation  programs. 
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Summary  of  Authority 

The  types  of  authority  exercised  by  inside  and  outside  evaluators 
are  quite  different.  Inside  evaluators  rely  on  respect  for  their 
knowledge  and  relationships  with  the  director  and  clinical  staff  in 
order  to  develop  their  evaluation  programs.  In  developing  their 
relationships  with  the  clinical  staff,  inside  evaluators  tend  to  avoid 
the  more  analytical  and  critical  types  of  evaluation. 

Outside  evaluators  rely  upon  their  formal  authority  and, 
especially,  their  control  over  the  purse  strings  of  the  organizations 
to  achieve  the  responsiveness  which  they  desire.  As  a consequence, 
they  do  not  have  to  "waste  time"  developing  "authority  of  person,  " 
but  the  exclusive  use  of  formal  authority  tends  to  provoke  reaction 
and  resistance  from  those  being  evaluated.  The  authority  of  competence 
of  outsiders  is  perceived  to  be  limited  either  because  of  their  lack 
of  substantive  knowledge  concerning  mental  health  services  or 
lack  of  time  and  adequate  methods  to  truly  understand  the  operations 
of  the  local  organizations. 

Information 

Evaluation  does  not  always  have  its  intended  impact.  It  is 
important  for  evaluators  to  possess  sufficient  information  to  judge 
whether  the  organizations  are  being  responsive  to  their  values  and 
priorities  as  a result  of  the  evaluations.  There  are  at  least  two 
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major  kinds  of  problems  in  utilizing  the  information  derived  from 
evaluation:  (1)  the  clarity  and  meaningfulness  of  the  data,  and  (2) 
the  accuracy  of  the  information. 

Inside  Evaluations 

It  has  already  been  described  how  inside  evaluation  is  generally 
introduced  in  such  a non -threatening  way  that  there  are  no  specific 
penalties  or  rewards  attached  to  compliance  with  it.  There  are 
also  no  systematic  plans  to  use  any  recommendations  or  implications 
that  could  be  drawn  from  the  evaluations  in  making  decisions  in  the 
organization.  Thus,  there  are  not  great  expectations  concerning 
the  anticipated  impact  of  these  inside  evaluations. 

However,  through  their  informal  contacts  with  the  clinical 
staff  and  their  personal  observation  of  the  operations  of  the  organiza- 
tions, perceptive  inside  evaluators  are  able  to  get  substantial  feed- 
back concerning  the  reaction  of  other  staff  members  to  their 
49 

evaluation.  Thus,  they  usually  have  at  least  a rough  estimate  of 
the  accuracy  of  the  data  and  its  meaningfulness  within  the  context 
of  the  particular  constraints  faced  by  the  individual  organization. 

49The  personality  of  the  evaluator  seemed  to  be  an  important 
factor  in  determining  the  amount  of  information  he  received  from  the 
clinicians.  While  the  sample  of  evaluators  is  quite  small,  certain 
evaluators  seemed  to  be  quite  gregarious  and  actively  sought  out 
feedback  from  the  clinical  staff.  Others  were  much  more  reserved 
and  appeared  to  have  less  contact  with  the  clinicians  and  less  feed- 
back as  a result. 
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Outside  Evaluations 


Much  of  the  data  provided  to  outsiders  is  seen  as  being  of 
doubtful  value  in  helping  them  to  determine  whether  the  organizations 
are  being  responsive  to  their  wishes.  One  district  board  requested 
a "program  budget"  from  a Center  as  a means  of  gaining  insight  into 
the  operations  of  the  Center  and  how  well  it  was  performing.  The 
evaluator  within  this  organization  questioned  whether  the  board  could 
adequately  interpret  the  data  from  this  budget.  He  perceives  that 
the  director  of  the  Center  is  attempting  to  "snow"  the  board  with 
the  data  provided  to  them: 

(Evaluator)  Unfortunately,  I think  that  he  would  like  to 
manipulate  evaluation  to  meet  his  needs  of  controlling  the 
board.  ...  I am  afraid  that  he  perceived  in  this  program 
budget.  . . his  real  tiirust  with  that  was  not  to  get  the  informa- 
tion because  he  [director]  hasn't  used  it.  His  thrust  was  to 
manipulate  the  board  with  that  data.  The  latest  thing  he  has 
asked  me  to  do  is  to  come  up  with  beautiful  charts  which  show 
the  highest  incidence  of  certain  types  of  social  indicators -- 
things  that  will  impress  the  board.  . I think  that,  again,  program 
evaluation  is  a camouflage  or  a diversion  for  the  board--to 
keep  the  board  interested,  busy,  and  out  of  his  hair. 

The  director  of  another  Center  is  frank  about  his  doubts  of  the 

board's  ability  to  understand  complex  data: 

(Director)  The  best  way  is  to  turn  out  statistics.  They 
[the  district  board]  generally  accept  that.  For  example,  how 
many  people  are  treated- -that  is  the  biggest  thing.  But  to  get 
down  any  further--I  just  don't  think  it  would  be  understood. 

It  is  not  just  the  lack  of  competence  which  prevents  outside 

boards  and  agencies  from  being  able  to  use  information  from 
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evaluations.  All  of  the  data  obtained  have  some  degree  of  ambiguity. 
A director  of  a Center  which  was  admittedly  very  high  on  the  cost 
per  patient  figures  is  quite  honest  in  describing  how  he  would  justify 
this  high  cost  to  the  district  board: 

(Director)  I am  going  to  go  in  there  with  some  fancy  foot- 
work and  razzamatazz,  and  say,  "Hey,  I have  sixty-seven 
inpatient  beds,  and  this  cost  data  means  less.  . . . Look  at 
all  of  our  positive  points.  . . . Well,  sure  [ name  of  another 
clinic]  is  doing  it  for  one  hundred  times  less  money  than  we 
are,  but  they  only  have  outpatient  services.  . . . And  [name 
of  another  mental  health  center  in  the  same  city]  have  fewer 
inpatient  beds,  and  they  don't  have  this,  and  we  have  this,  and 
we  have  to  do  this  and  that.  . . . " So  when  it  comes  down  to 
this  dollars  and  cents  thing,  they  are  going  to  be  confused 
anyway.  So  ultimately  it  will  go  back  to --have  we  impressed 
them  with  what  we  are  doing?  And  if  we  have  done  that,  we  are 
going  to  survive. 

As  far  as  the  state  and  national  agencies  are  concerned,  a 
great  problem  is  the  extent  to  which  their  "evaluations"  lead  to 
goal  displacement.  One  director  points  out  that  decreases  in 
hospitalization  rates  can  be  achieved  in  several  ways: 

(Director)  I am  concerned  with  the  pre-existing  conclusions. 
When  lack  of  hospitalization  is  considered  as  a criteria  of 
success,  you  can  tell  people  if  they  ever  show  up  in  the  county 
again  you  are  going  to  put  them  in  the  hospital  forever. 

Or  the  problem  of  high  inpatient  figures  can  be  "solved"  by  drugs: 

(Director)  If  you  keep  a patient  quiet  with  2200  milligrams 
of  thorazine  and  half  of  them  die,  and  you  didn't  have  to  admit 
any  of  them,  you  have  gotten  great  results.  ...  Or,  if  they 
don't  die,  they  are  likely  to  be  zombie -like.  . . . 

The  accuracy  of  the  data  collected  in  evaluations  is  questionable 

for  several  reasons.  Sometimes  the  organizations  simply  do  not 
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possess  the  information  requested,  as  in  the  following  case: 

(Director)  I had  a hard  time  filling  out  the  financial  report 
because  our  hospital  does  not  break  down  the  Center  as  a 
separate  entity.  So  when  I get  together  with  the  financial  director 
for  the  hospital  and  say,  "Well,  they  want  to  know  what  our 
expenses  are  for  this  fiscal  year?  "--he  doesn't  know.  I 
mean  with  a 100  percent  certainty,  he  does  not  know  what  it  is. 

The  staff  members  in  one  organization  were  sharply  criticizing 

the  paperwork  required  by  outside  agencies.  However,  one  staff 

member  told  how  she  was  able  to  fill  out  a complex  form  in  a few 

minutes  by  giving  short  and  minimal  replies  to  questions.  The 

director  of  an  organization  strongly  committed  to  indirect  services 

told  the  author  how  he  "handled"  forms  which  emphasize  large 

numbers  of  direct  encounters  as  a criterion  of  "goodness.  " He 

said  he  had  developed  the  concept  of  "non-case"  encounters  which 
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he  included  in  the  figures  used  for  calculating  the  cost  per  encounter. 


^Another  member  of  this  organization  who  received  the  forms 
sent  by  this  staff  member  confirmed  her  contention  about  the  little 
time  spent  on  the  forms,  but  said  that,  as  a result,  it  was  impossible 
to  acquire  needed  data  about  the  patient  from  the  form. 
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However,  this  director  admitted  that  the  latest  form  of  account- 
ability developed  by  NIMH  virtually  excluded  indirect  services.  He 
noted  that  it  was  almost  "as  if  they  want  you  to  fail.  " This  points  out 
the  fact  that  the  agencies  were  able  to  be  quite  critical  of  individual 
mental  health  centers  despite  the  doubts  expressed  on  this  point  by 
private  psychiatrists.  The  reason  for  their  detachment  from  the  cause 
of  the  local  centers  is  the  fact  that  the  survival  of  the  state  and  national 
agencies  is  becoming  more  and  more  tied  to  a research  and  evaluation 
role.  NIMH  is  encouraging  community  evaluation  of  the  mental  health 
centers  and  offering  consultation  to  the  community  groups  in  order  to 
help  them  carry  out  such  evaluations.  See  Susan  Salas  in,  "Conversational 
Contact  with  Bertram  S.  Brown,  " Evaluation,  Vol.  1,  No.  2 (1973), 
p.  17. 
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Thus,  lack  of  accuracy  in  the  information  can  result  from  lack  of 

availability  of  the  information,  inadequate  attention  to  the  forms  by 

staff  members,  and  "interpretation"  of  the  data  which  deviates 

from  its  original  interpretation  by  outside  agencies. 

Another  form  of  inaccuracy  is  the  result  of  deliberate  cheating. 

Indeed,  many  of  the  persons  within  the  organization  freely  admit 

that  cheating  occurs.  Unlike  the  cheating  described  by  Cohen,  ^ 

much  of  the  cheating  on  forms  is  seen  as  contributing  to  the 

accomplishment  of  organizational  goals  by  cutting  down  on  what  the 
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staff  regard  as  unreasonable  demands  made  by  outside  agencies. 

A director  of  one  organization  describes  how  one  of  the  workers  on 
his  staff  became  skeptical  of  the  amount  of  data  required  by  the 
state  agency: 

(Director)  First  she  fabricated  the  data.  Then  she  stopped 
sending  it  in  altogether.  She  said,  "Nobody  can  possibly  want 
this.  " It  took  the  State  nine  months  to  find  out  what  had  happened. 


^Harry  Cohen  has  done  one  of  the  few  systematic  studies  of 
this  widespread  and  important  behavior  pattern  in  organizations -- 
cheating.  See  Harry  Cohen,  The  Demonics  of  Bureaucracy  (Ames: 
Iowa  State  University  Press,  1965). 
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Thus,  one  clinician  described  to  the  author  how  it  was 
frustrating  for  a client  to  come  in  with  a desire  to  discuss  his 
problems  immediately  and  then  to  be  faced  with  answering  questions 
on  an  intake  form  required  by  the  State.  This  clinician  attempted 
to  avoid  "putting  off"  the  patient  by  letting  him  discuss  his  problems 
and  obtaining  as  much  information  required  by  the  intake  form  as 
possible  from  this  discussion. 


232 


A clinician  at  one  Center  directly  told  the  director  and  evaluator 
in  the  organization  that  it  "was  easy  to  make  it  look  like  you  are 
doing  a lot"  with  the  activity  sheets  used  by  organizations. 

Another  clinician  told  a state  official  involved  in  introducing  evalua- 
tion that  "if  we  have  to  put  down  numbers,  we  are  going  to  cheat.  " 

Summary  of  Information 

Merely  carrying  out  an  evaluation  by  no  means  guarantees  that 
the  staff  will  actually  be  responsive  to  the  values  which  the  evaluator 
wishes  to  emphsize.  Moreover,  the  data  obtained  by  the  evaluation 
will  not  necessarily  be  clear  or  accurate. 

Inside  evaluators  begin  with  much  lower  expectations  than  out- 
side evaluators  as  to  what  their  evaluations  are  likely  to  accomplish. 
However,  their  informal  contacts  with  the  staff  and  ability  to  observe 
the  behavior  of  the  organizational  members  give  them  a rough  idea 
of  the  accuracy  and  meaningfulness  of  the  data  obtained. 

Outside  evaluators  are  likely  to  have  greater  difficulty  in 
interpreting  the  meaningfulness  of  the  data  and  its  degree  of  accuracy. 
They  have  to  contend  with  ambiguity  of  interpretation,  information 
overload,  and  lack  of  accuracy  in  the  data.  Moreover,  the  staff 
whom  these  outsiders  are  evaluating  are  not  apologetic  about  the 
cheating  and  "curt"  attention  given  to  demands  for  evaluation  and 
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accountability. 
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Conclusions 

The  purpose  of  this  chapter  was  to  examine  the  relationship 
between  evaluation,  accountability,  and  responsibility.  In  particular, 
the  factors  which  are  important  in  determining  the  degree  of  responsive- 
ness which  an  evaluator  can  achieve  from  those  whom  he  evaluates 
were  analyzed.  To  aid  in  this  analysis,  comparisons  between  insider 
and  outsider  evaluations  were  utilized.  Four  problem  areas  are  seen 
as  being  important  in  determining  the  amount  of  responsiveness  to 
evaluators  and  evaluation:  (1)  the  kinds  of  values  held  by  the  evalua- 
tors and  persons  being  evaluated,  (2)  the  motivation  behind  the 
evaluation --its  nature  and  strength,  (3)  the  kinds  of  authority 
possessed  and  exercised  by  the  evaluators  and  the  reaction  of  those 
being  evaluated  to  them,  and  (4)  the  extent  to  which  the  evaluators 
possess  the  information  necessary  to  discover  whether  the  evalua- 
tion is  achieving  its  intended  impact. 
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The  attitudes  of  these  staff  members  bear  striking  resemblance 
to  the  attitudes  of  the  personnel  in  "OPA.  " Victor  Thompson  writes, 

"At  the  end  of  rationing,  some  branch  personnel  thought  that  they 
spent  so  much  time  explaining  to  their  superiors  what  they  did.  . . 
that  they  did  not  have  much  time  left  to  do  anything.  " He  also  notes 
that  "the  relationship  tended  to  develop  into  one  where  the  working 
people  seemed  to  be  mischievious  children  who  had  to  be  controlled 
rather  than  experts  who  were  actually  responsible  for  the  successful 
rationing  of  the  commodity.  " See  Victor  A.  Thompson,  The  Regulatory 
Process  in  OPA  Rationing  (New  York:  King's  Crown  Press,  1950), 
p.  231. 
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It  was  found  that  the  degree  of  congruence  between  the  values 
of  the  evaluators  and  those  being  evaluated  is  important  in  determining 
the  need  for  evaluation.  Evaluation  for  the  purposes  of  accountability 
and  responsibility  is  likely  to  be  less  necessary  when  the  persons 
being  evaluated  share  the  values  of  the  evaluators.  When  the  two 
value  systems  diverge,  there  is  likely  to  be  reaction  and  resistance 
to  the  evaluation.  Another  problem  is  that  the  values  and  goals  of 
persons,  programs,  agencies  and  mental  health  organizations  in 
general  are  not  very  clear  once  one  gets  beyond  gross  generalities. 

In  such  a situation  there  is  a great  chance  for  goal  displacement  in 
which  the  staffs  of  the  organizations  overemphasize  goals  included 
in  the  forms  required  by  funding  agencies.  Indeed,  in  the  absence 
of  clearly  stated  and  operational  goals,  the  "forms  of  accountability" 
are  seen  by  the  personnel  of  the  organizations  as  the  values  desired 
by  the  management  (for  internal  evaluations)  or  outside  agencies 
and  district  boards  (for  external  evaluations). 

There  is  evidence  that  persons  outside  of  the  organization  tend 
to  emphasize  monetary  and  quantitative  indices  while  internal 
evaluations  tend  to  be  more  concerned  with  problems  of  the  quality 
of  service  being  provided.  When  persons  being  evaluated  sharply 
disagree  with  the  values  emphasized  by  the  evaluation  and  forms  of 
accountability,  they  may  see  evaluation,  accountability,  and  responsi- 
bility as  actually  being  "negative  values"  and  actively  resist  them. 
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The  motivation  to  evaluate  is  the  second  element  in  our  analysis 
of  the  problem  of  achieving  responsibility.  Outside  agencies  and 
boards  tend  to  see  evaluation  as  having  the  purpose  of  "catching  the 
cheaters"  and  achieving  the  accountability  and  responsiveness  which 
they  desire  from  the  organizations  being  evaluated.  Inside  evaluators 
and  other  staff  in  these  organizations  prefer  to  use  evaluation  as  a 
means  of  providing  internal  feedback  rather  than  using  it  primarily 
for  the  purpose  of  forcing  responsiveness  to  community  and  other 
"outsider " values.  Thus,  inside  evaluations  tend  to  be  less  "critical" 
in  the  sense  that  they  are  not  usually  intended  to  prove  whether  or 
not  the  programs  are  worth  continuing  because  this  is  assumed  to 
be  the  case. 

Nevertheless,  it  was  found  that  there  are  several  factors  which 
do  facilitate  highly  critical  types  of  internal  evaluations.  However, 
the  most  critical  forms  of  evaluation  are  not  the  formalized  evalua- 
tions, which  are  expected  to  be  highly  positive  in  nature,  but  the 
more  informal  and  subjective  types  of  judgments  that  constantly 
take  place  in  organizations.  Professionalism  does  not  prove  to 
be  a major  obstacle  to  the  implementation  of  formal  evaluation  and 
probably  contributes  in  part  to  the  impetus  for  critical  internal 
evaluation.  Generally,  highly  critical  attitudes  are  correlated  with 
various  factors  which  made  the  evaluator  feel  apart  or  "removed  from" 
the  persons,  programs,  and  organizations  which  he  is  evaluating.  This 
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detachment  can  result  from  being  in  a different  program  or  level 
of  hierarchy  in  the  organization  than  those  being  evaluated.  It 
can  also  result  from  differences  in  values  from  those  being  evaluated 
or  career  ambitions  which  would  take  the  evaluator  away  from  the 
organization  and  those  being  evaluated. 

In  order  to  carry  out  an  evaluation,  an  evaluator  must  possess 
some  kind  of  authority.  Three  major  types  are  singled  out: 

(1)  authority  of  position,  (2)  authority  of  person,  and  (3)  authority 
of  competence.  Authority  of  person  is  subdivided  into  three  com- 
ponents of  the  evaluator's  relationships  and  duties  in  the  organiza- 
tion: his  relationship  to  the  director  and  clinical  staff,  and  his 
performance  of  non -evaluative  tasks.  Questionnaire  data  indicates 
that  technical  competence  is  regarded  as  the  single  most  important 
factor,  and  formal  power  as  being  the  least  important  by  the  persons 
sampled. 

However,  interviews  and  participant  observation  data  indicate  that 
the  evaluator's  role  is  a particularly  precarious  one.  He  needs  to 
develop  all  potential  sources  of  authority.  His  relationships  with 
the  director  and  clinical  staff  are  also  critical  factors  in  determining 
whether  he  is  effective.  Ironically,  the  evaluator's  attempts  to 
develop  these  relationships,  especially  those  with  the  clinical  staff, 
tend  to  deflect  him  from  his  original  task  of  evaluation  towards 
mere  data  gathering  and  description. 
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Outside  evaluators  rely  almost  totally  on  their  formal  authority 
or  control  over  the  purse  strings.  The  use  of  formal  authority 
exclusively  tends  to  evoke  reaction  and  resistance  from  organizational 
members.  Outsiders  are  not  recognized  as  having  authority  of 
competence  to  the  same  extent  as  inside  evaluators  in  these  organiza- 
tions. The  district  boards  lack  knowledge  concerning  the  quality  of 
mental  health  programs  and  methods.  Agency  members  are  seen 
as  having  a greater  degree  of  knowledge  of  mental  health  programs 
but  as  lacking  the  time  and  motivation  to  learn  about  the  particular 
context  and  constraints  of  specific  mental  health  organizations. 

The  final  factor  in  determining  the  amount  of  responsiveness 
which  an  evaluator  can  achieve  is  the  amount  of  knowledge  or  informa- 
tion concerning  whether  the  forms  of  evaluation  and  accountability 
actually  have  their  intended  impact.  The  major  problem  of  the  inside 
evaluators  is  that  they  have  designed  their  evaluations  mainly  to 
provide  feedback  without  any  systematic  check  to  find  out  whether 
evaluation  has  any  impact.  However,  they  are  usually  able  to 
secure  substantial  feedback  concerning  the  effect  of  evaluation 
through  observation  of  the  organization's  activities  and  their 
relationship  with  the  clinical  staff  of  the  organization. 

Outsiders  rely  mainly  on  data  provided  to  them  by  people  within 
the  organizations  and  on-site  visits.  The  data  is  often  inaccurate 
because  of  the  fact  that  it  is  perfunctorily  collected  or  because 
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accurate  estimates  for  the  figures  are  not  actually  available.  Also, 
outsiders  are  seen  as  frequently  not  able  to  digest  or  interpret  these 
figures  meaningfully.  Finally,  there  are  significant  chances  for 
deliberate  misrepresentation  of  the  data  by  the  staff  of  the  organiza- 
tions. Cheating  does  not  appear  to  be  done  so  much  for  personal 
gain  or  advantage  of  the  organization  as  it  is  for  limiting  the  amount 
of  time  spent  on  accountability  and  evaluation. 

More  briefly,  the  arguments  above  may  be  summarized  in  the 
following  form: 

(1)  The  degree  of  responsiveness  which  an  evaluator  achieves 
to  the  values  which  he  wishes  to  emphasize  is  a positive  function 
of:  (a)  the  amount  of  authority  he  possesses,  (b)  the  degree  of 
his  motivation  to  achieve  this  responsiveness,  and  (c)  the 
extent  to  which  he  possesses  accurate  information  concerning  the 
actual  impact  of  the  evaluation. 

(2)  The  degree  of  responsiveness  which  an  evaluator  achieves 
to  the  values  which  he  desires  to  emphasize  is  a negative 
function  of:  (a)  the  difference  between  his  own  values  and  the 
values  of  those  whom  he  is  evaluating,  (b)  the  degree  of  goal 
displacement  which  occurs,  and  (c)  the  degree  of  reaction 
against  his  use  of  authority. 

Comparisons  were  utilized  between  inside  and  outside  evaluators 
concerning  the  factors  which  facilitated  or  hindered  them  from 
achieving  responsiveness  to  their  values.  It  was  found  (not 
unsurprisingly)  that  each  has  its  advantages  in  achieving  this 
responsiveness.  Outsiders  have  greater  formal  authority  and  can 
afford  to  be  much  more  critical  in  the  nature  of  their  evaluations. 
Insiders  have  the  capacity  to  gather  more  accurate  information 
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concerning  whether  the  desired  impact  has  actually  been  achieved. 

They  also  evoke  less  reaction  from  the  staff.  Organizations  generally 
do  not  willingly  provide  information  to  outsiders  that  reflects 
negatively  on  them.  Internal  evaluations  may  or  may  not  provide  the 
functional  equivalent  of  outside  evaluations.  Whether  they  do  or  not 
depends  greatly  on  the  degree  of  congruence  between  the  values  and 
priorities  which  inside  and  outside  evaluators  are  interested  in 
emphasizing. 

Accountability  and  responsibility  are  strongly  tied  in  with 
value  judgments.  Moreover,  they  are  generally  seen  as  being 
positive  values  by  the  public  and  people  within  the  organizations. 
However,  they  are  achieved  at  a cost  of  time  spent  collecting  the 
information,  and  also  the  anxiety  and  reaction  aroused  as  a result 
of  the  motivation  which  the  staff  perceived  behind  them--a  lack  of 
trust. 


CHAPTER  VI 


SUMMARY  AND  CONCLUSIONS 

The  major  aim  of  this  study  has  been  to  analyze  the  role  that 
formal,  scientific  evaluation  has  played  in  the  mental  health 
organizations.  Formal  evaluation  was  defined  as  "the  process  of 
measuring  goals,  however  defined,  through  the  use  of  the  tools  of 
science.  " In  particular,  the  focus  of  this  study  was  the  extent  to 
which  organizations  can  critically  evaluate  their  own  performance. 
Three  dimensions  of  formal  evaluation  were  singled  out  in  order 
to  distinguish  between  tonen  and  critical  evaluations:  scope,  focus, 
and  substance. 

The  sample  of  organizations  in  this  study  included  fourteen 
mental  health  organizations  in  the  State  of  Florida.  Three  major 
kinds  of  data  were  used:  (a)  questionnaires  to  certain  clinicians, 
directors,  and  evaluators  in  these  fourteen  mental  health  organizations, 

(b)  interviews  with  individuals  from  these  same  three  groups,  and 

(c)  approximately  five  months  of  participant  observation  in  one 
organization. 

The  problem  of  what  role  evaluation  has  played  in  this  group 
of  organizations  was  seen  as  being  related  to  four  areas  of 
organizational  and  public  administration  research:  (1)  the  relationship 
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between  the  natural  and  artificial  systems  of  an  organization,  (2) 
the  technology  employed  by  organizations,  (3)  the  problem  of  achieving 
accountability  and  responsibility  in  public  organizations,  and 
(4)  the  relationship  between  survival  and  goal -attainment  models 
of  organizations.  Insights  from  the  literature  of  organizational 
theory  and  public  administration  concerning  these  problem  areas 
were  employed  in  our  analysis  of  the  role  of  evaluation. 

In  Chapter  II,  the  technology  and  impact  of  formal  evaluation 
in  these  fourteen  organizations  were  examined  with  respect  to  the 
dimensions  of  scope,  focus,  and  substance.  The  major  finding  was 
that  formal  evaluation  had  little  visible  impact  on  the  operations 
of  the  organizations  in  which  it  was  introduced.  At  the  time  of 
this  study,  evaluation  was  being  used  internally  to  some  degree  for 
the  purpose  of  providing  feedback  to  the  personnel  of  the  organiza- 
tions concerning  the  reactions  of  the  clientele  to  the  services  which 
they  received. 

In  Chapter  III,  attitudes  toward  formal  evaluation  were  analyzed. 
Questionnaire  data  revealed  a favorable  orientation  toward  formal 
evaluation  for  general  purposes  among  the  clinicians,  directors,  and 
evaluators  surveyed.  Not  surprisingly,  clinicians  favored  more 
generalized  uses  of  evaluation  to  more  specific  and  potentially 
threatening  uses.  Less  expectedly,  the  evaluators  preferred  the 
same  types  of  non -threatening  uses  of  evaluation.  They  apparently 
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had  encountered  little  direct  opposition  because  they  favored  such 
non -threatening  uses  of  evaluation.  Thus,  the  most  important 
effect  of  the  natural  system  was  seen  as  being  indirect  because  the 
evaluators  had  anticipated  the  reactions  of  others  in  the  organization 
to  threatening  evaluations.  Two  inductive  hypotheses  were  suggested 
concerning  attitudes  towards  evaluations: 

(1)  Persons  being  evaluated  will  favor  those  evaluations 
which  minimize  potential  threats  to  themselves  and  maximize 
potential  benefits. 

(2)  The  degree  to  which  evaluators  favor  non -threatening 
evaluations  will  be  a positive  function  of  the  amount  of  resistance 
anticipated  by  the  evaluator  and  a negative  function  of  the  amount 
of  authority  possessed  by  the  evaluator. 

The  attitudes  of  clinicians  and  directors  in  the  organizations 
under  study  toward  the  technology  of  evaluation  were  ambivalent. 

They  were  willing  to  experiment  with  formal  evaluation  but  retained 
substantial  doubt  concerning  its  validity  and  meaningfulness  for 
making  specific  decisions  or  comparisons.  Clinicians  and  directors 
generally  rejected  attempts  to  judge  the  worth  of  their  activities 
and  programs  that  ignored  the  particular  constraints  which  they 
faced.  The  preference  for  program  evaluation  was  significant  in 
view  of  the  fact  that  the  most  prevalent  forms  of  evaluation  employed 
by  these  organizations  were  actually  gauged  to  measuring  individual 
performances.  Thus,  the  kinds  of  evaluations  favored  were  not 
controlled  by  the  kinds  of  techniques  available. 
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The  actual  decision  making  process  of  these  organizations 
was  studied  with  respect  to  evaluation.  It  was  found  that  evaluative 
judgments  were  constantly  being  made.  However,  these  judgments 
were  mainly  of  an  informal  and  highly  subjective  nature.  They  were 
based  on  the  eclectic  combination  of  personal  values  of  decision 
makers  and  whatever  sources  of  information  that  were  available 
to  them.  The  nature  of  the  decision  making  process  of  these  organiza- 
tions made  it  appear  that  formal  evaluation  would  play  a marginal 
(though  potentially  significant)  role  in  this  process. 

All  evaluations,  formal  or  informal,  emphasize  certain  values. 

The  implementation  of  formal  evaluation  is  likely  to  make  the  values 
behind  the  evaluation  more  apparent.  Where  values  held  by  organiza- 
tional members  differ  substantially,  formal  evaluation  can  potentially 
increase  conflict. 

The  reasons  why  many  of  these  organizations  had  decided  to 
invest  in  formal  evaluation  was  studied.  It  was  noted  that  many 
organizations  were  hesitant  to  invest  in  formal  evaluation  and 
evaluators  because  of  the  actual  or  potential  costs  involved.  However, 
those  organizations  which  did  invest  in  evaluation  expect  the  follow- 
ing actual  or  potential  benefits  to  result  from  investment  in  it: 

(1)  giving  the  organization  a headstart  on  future  evaluation  that  is 
likely  to  be  required,  (2)  providing  the  organization  with  a method  of 
justifying  the  public  funds  being  spent  in  support  of  it,  (3)  yielding 
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dollars  and  cents  benefits  in  the  form  of  grants  that  were  tied 
directly  or  indirectly  to  the  performance  of  formal  evaluation, 
and  (4)  the  contribution  of  the  evaluation  staff  to  the  performance 
of  non -evaluative  tasks  crucial  or  important  to  the  survival  of  the 
organization.  A hypothesis  was  developed  to  explain  the  relatively 
great  interest  of  mental  health  organizations  in  undertaking  formal 
evaluation: 

(3)  The  degree  to  which  an  organization  attempts  to 
anticipate  its  environment  is  a function  of  the  interaction  between 
the  degree  of  concern  of  the  organizational  members  for  its 
survival  and  the  extent  to  which  organizational  members  are 
capable  of  scanning  the  environment. 

The  reasons  for  the  source  of  variation  in  the  extent  to  which 
these  organizations  invested  in  formal  evaluation  were  sought. 

It  was  found  that  by  far  the  most  important  variable  was  the  attitude 
of  the  director  of  the  organization.  Directors  weighed  the  advantages 
and  disadvantages  associated  with  undertaking  formal  evaluation 
for  the  survival  of  the  organization.  They  also  considered  the  merits 
of  evaluation  with  respect  to  their  personal  values  and  ambitions. 

There  was  an  apparent  discrepancy  between  the  preferences 
expressed  by  the  clinicians  on  the  questionnaire  concerning  the 
functions  of  evaluation  and  the  purposes  actually  behind  the  impetus 
to  undertake  formal  evaluation.  It  was  also  noted  how  evaluators 
became  more  involved  in  "non -evaluative " tasks  than  in  their 
internal,  formal  evaluation  tasks.  Two  hypotheses  were  developed 
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to  explain  the  above  occurrences: 

(4)  Evaluation  activities  associated  with  organizational 
survival  will  displace  other  activities  which  are  not  directly- 
related  to  organizational  survival. 

(5)  The  degree  of  displacement  of  evaluation  from  goal- 
oriented  objectives  to  survival  objectives  is  a function  of  the 
degree  of  concern  for  organizational  survival  of  organizational 
members  and  the  number  of  demands  made  by  outside  agencies 
on  these  organizations. 

In  Chapter  V,  the  relationship  between  evaluation  and  the  concepts 
of  accountability  and  responsibility  was  analyzed.  A comparison  was 
utilized  between  inside  and  outside  evaluations  in  order  to  study  the 
factors  important  to  an  evaluator  in  the  achievement  of  responsiveness 
to  his  evaluation.  Four  major  concepts  were  seen  as  being  the  key 
to  understanding  the  degree  of  responsiveness  achieved:  values, 
motivation,  authority,  and  information.  Two  hypotheses  were 
developed  which  summarized  the  relationships  among  these  factors 
(and  subcomponents  of  them): 

(6)  The  degree  of  responsiveness  which  an  evaluator  achieves 
to  the  values  which  he  wishes  to  emphasize  is  a positive  function 
of:  (a)  the  amount  of  authority  he  possesses,  (b)  the  degree  of 
his  motivation  to  achieve  this  responsiveness,  and  (c)  the 
extent  to  which  he  possesses  accurate  information  concerning  the 
actual  impact  of  the  evaluation. 

(7)  The  degree  of  responsiveness  which  an  evaluator  achieves 
to  the  values  which  he  desires  to  emphasize  is  a negative 
function  of:  (a)  the  difference  between  his  own  values  and  the 
values  of  those  whom  he  is  evaluating,  (b)  the  degree  of  goal 
displacement  which  occurs,  and  (c)  the  degree  of  reaction 
against  his  use  of  authority. 

At  the  very  beginning  of  this  study,  it  was  noted  that  the 
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original  inspiration  for  this  study  was  an  article  by  Aaron  Wildavsky 
which  suggested  that  organizations  may  be  incapable  of  evaluating 
themselves.  ^ In  hypothetical  form,  Wildavsky's  hypothesis  was 
stated  as  follows: 

Due  to  survival  needs,  organizations  cannot  critically 
evaluate  themselves,  and  any  evaluations  which  they  do 
implement  will  tend  to  be  narrow,  technique -oriented,  and 
symbolic. 

It  was  argued  in  Chapter  IV  that  the  major  reason  behind  the 
movement  towards  formal  evaluation  was  the  organization's 
struggle  for  survival.  The  apparent  contradiction  of  this  conclusion 
with  the  Wildavsky  hypothesis  is  explained  by  the  nature  of  the  formal 
evaluations  undertaken  by  these  organizations.  They  had  introduced 
evaluation  in  such  a non -threatening  manner  that  there  was  little 
evidence  of  resulting  substantive  change.  Evaluation  was  expected 
to  contribute  to  the  survival  of  the  organizations  which  invested  in 
it.  Based  upon  the  way  that  evaluation  and  evaluators  were  dis- 
placed toward  survival  purposes  and  activities,  it  does  appear  that 
organizations  cannot  closely  approach  what  Wildavsky  calls  a 
perfectly  self-evaluating  organization. 

Nevertheless,  critical  judgments  and  evaluation  did  take  place 
within  the  organizations.  Many  of  the  persons  in  these  organizations 

■^Aaron  Wildavsky,  'The  Self-Evaluating  Organization,  " Public 
Administration  Review,  Vol.  32  (November/December,  1972), 
pp.  509-520. 
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did  appear  to  be  quite  interested  in  receiving  feedback  from  the 
formal  evaluation.  The  increment  of  knowledge  added  by  evaluation 
may  be  important  for  directors  and  clinicians  in  making  some 
decisions.  Moreover,  there  are  several  factors  which  enable 
individuals  to  critically  examine  the  worth  of  the  services  provided 
by  the  organization,  such  as  values  and  ambitions  brought  in  from 
outside  the  organization. 

Perhaps  an  even  more  important  limitation  than  the  internal 
ones  upon  the  possibility  of  truly  critical  self-evaluation  by 
organizations  is  the  nature  of  these  organization's  environments. 
Outsiders  assumed  that  the  organizations  would  make  themselves 
look  as  good  as  possible.  Members  of  these  organizations  perceived 
this  expectation  and  generally  attempted  to  fulfill  it  by  presenting 
information  to  outsiders  which  reflected  favorably  on  their  organiza- 
tions. The  extent  to  which  critical,  internal  evaluation  (formal 
or  informal)  provides  the  functional  equivalent  of  outside  evaluation 
depends  upon  the  degree  of  congruence  between  the  values  and 
priorities  of  insiders  and  outsiders. 

Based  upon  the  analysis  above,  there  would  appear  to  be  two 
fairly  effective  ways  to  facilitate  critical  organizational  self- 
evaluation.  One  way  is  by  rewarding  individuals  within  the  organiza- 
tions (and  organizations  as  a whole)  for  reporting  accurate  rather  than 
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2 

merely  favorable  information.  A second  way  is  to  encourage  the 
growth  of  value  systems  and  attachments  which  are  more  cosmo- 

O 

politan,  less  local.  However,  it  should  be  noted  that  increased 
ability  of  organizations  to  evaluate  themselves  is  likely  to  have 
many  costs,  e.  g. , instability. 

In  summary,  very  few  organizations,  if  any,  are  capable  of 
fulfilling  Wildavsky's  model  of  a self-evaluation  organization. 
However,  organizations  are  open  systems  and  absorb  elements 
which  do  facilitate  self-evaluation.  However,  evaluation  appears 
to  be  one  of  the  most  difficult  activities  to  carry  out  in  organiza- 
tions because  of  the  tremendous  anxiety  and  other  costs  associated 
with  it.  While  internal  evaluators  appeared  to  be  making  very  slow 
headway  as  far  as  accomplishing  visible  change  is  concerned,  they 
appeared  in  a much  more  favorable  light  when  compared  to  outside 
evaluators.  The  latter  often  appear  to  generate  a negative  effect 
upon  the  organizations  that  they  evaluate. 

Evaluation  may  or  may  not  achieve  the  goals  intended  to  be 
accomplished  by  it.  It  may  or  may  not  be  worth  the  cost  of  the 

2 

Wildavsky  pointed  out  that  trust  is  a condition  of  successful 
evaluation  in  Wildavsky,  ibid.  , pp.  519-520. 

O 

For  a recent  attempt  to  encourage  "cosmopolitan"  attach- 
ments in  organizations,  see  Ralph  Nadar,  Peter  J.  Petkas,  and  Kate 
Blackwell,  eds..  Whistle  Blowing  (New  York:  Grossman  Publishers, 
1972). 
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resources  invested  in  it.  In  short,  evaluation  itself  needs  to  be 
evaluated.  Renos  Dubos  has  summarized  the  position  taken  here 
quite  well: 

What  is  new  is  not  necessarily  good,  and  all  changes,  even 
those  apparently  most  desirable,  are  always  fraught  with 
unpredictable  consequences.  The  scientist  must  beware  of 
having  to  admit,  like  Captain  Ahab,  in  Melville's  Moby  Dick, 
"All  my  means  are  sane;  my  motives  and  objects  mad. 


4 

Renos  Dubos,  Mirage  of  Health:  Utopias,  Progress,  and 
Biological  Change  (New  York:  Harper  and  Row,  1959),  pp.  227-228. 


APPENDIX 


The  Questionnaire 


Note:  Evaluation  below  refers  specifically  to  formal  evaluation 

carried  out  by  an  evaluator  and  his  staff  (in  a mental  health  organi- 
zation). 

1.  Evaluation  can  and  should  be  used  to  furnish  feedback  for  the 
clinical  staff. 

Strongly  Agree 

Agree 

Undecided 

Disagree 

Strongly  Disagree 

(The  identical  series  of  five  response  categories  is  repeated  after 

each  item. ) 

2.  Evaluation  can  and  should  be  used  to  determine  whether 
individual  mental  health  programs  are  successful  or  not. 

3.  Evaluation  can  and  should  be  used  to  determine  whether  or 
not  the  overall  organization  is  performing  well. 

4.  It  is  likely  that  evaluation  will  cost  more  than  the  information 
derived  from  it  is  worth. 

5.  Only  the  clinical  staff  involved  in  direct  services  can  actually 
judge  whether  or  not  programs  are  successful. 

6.  Evaluation  can  and  should  be  used  to  determine  whether  or 
not  individual  clinical  staff  members  are  performing  their 
jobs  well. 
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7.  Evaluation  should  only  be  undertaken  after  the  mental  health 
organization  has  met  the  demands  for  resources  by  the  direct 
services. 

8.  Evaluation  can  and  should  be  used  to  provide  accountability 
of  the  mental  health  organization  to  the  community  it  serves. 

9.  Evaluation  can  and  should  be  used  to  provide  accountability 
of  the  clinical  staff  to  the  clientele  of  the  mental  health 
organization. 

10.  Evaluation  is  unlikely  to  add  to  our  knowledge  concerning  the 
dynamics  of  mental  health. 

11.  Evaluation  can  and  should  be  used  to  determine  who  should  be 
fired  from  their  jobs. 

12.  Evaluation  is  likely  to  be  used  as  a means  of  control  over  the  staff. 

13.  Staff  are  likely  to  overemphasize  the  activities  and  goals 
measured  by  evaluation. 

14.  It  is  impossible  to  measure  quantitatively  the  most  important 
effects  of  mental  health  direct  services. 

15.  Evaluation  can  and  should  be  used  to  determine  how  much  funding 
mental  health  organizations  should  receive. 

16.  Evaluation  can  and  should  be  used  to  determine  which  service 
programs,  if  any,  are  performing  poorly. 

17.  Evaluation  is  unlikely  to  produce  information  relevant  to  the 
people  performing  direct  services. 

18.  Evaluation  is  currently  neither  valid  nor  reliable  enough  to 
be  used  to  determine  who  should  be  promoted  within  the 
organization. 

19.  Evaluation  is  likely  to  increase  the  efficiency  of  the  organization. 

20.  Evaluation  can  and  should  be  used  to  determine  the  impact  of 
the  organization's  programs  on  the  community. 

21.  Evaluation  is  likely  to  improve  the  quality  of  administrative 
decision  making  in  the  organization. 
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22.  Evaluation  is  neither  valid  nor  reliable  enough  to  be  used  to 
determine  who  is  doing  a good  job. 

23.  Most  of  the  information  derived  from  evaluation  is  unlikely 
to  be  used  in  actual  administrative  decision  making. 

24.  Evaluation  is  necessarily  of  lower  priority  than  the  direct 
delivery  of  services. 

25.  Professional  clinical  staff  are  capable  of  evaluating  their  own 
performance  without  the  aid  of  a formal  evaluation  by  an 
evaluator. 

26.  Evaluation  is  likely  to  be  used  to  regulate  the  behavior  of  the 
clinical  staff. 

27.  Community  control  over  mental  health  services  can  and  should 
be  increased  through  the  use  of  evaluation. 

28.  A higher  percentage  of  the  total  budget  (of  the  mental  health 
organization)  should  be  devoted  to  evaluation  than  is  currently 
the  case. 

Please  rank  the  following  factors  in  what  you  regard  to  be  their 
order  of  importance  as  oostacies  to  evaluation.  Evaluation  below 
refers  specifically  to  formal  evaluation  carried  out  by  an  evaluator 
and  his  staff.  (1  = the  most  important  obstacle;  5 = the  least  important 
obstacle). 


The  lack  of  reliability  and  validity  of  the  current 
evaluation  techniques. 

Anxiety  on  the  part  of  the  direct  service  staff  as 
to  how  evaluation  is  to  be  used. 

The  belief  of  professionals  that  they  should  evaluate 
themselves  and  don't  need  formal  evaluation  by  an 
evaluator. 

The  lack  of  skills  necessary  in  the  evaluation  staff 
to  carry  out  a valid  and  reliable  evaluation. 

The  lack  of  resources  of  time,  money,  and  staff 
for  the  researchers  to  carry  out  valid  and  reliable 
evaluations. 
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Please  rank  the  following  in  what  you  regard  to  be  their  order 
of  importance  as  functions  of  evaluation.  Evaluation  below  refers 
specifically  to  formal  evaluation  carried  out  by  an  evaluator  and  his 
staff.  (1  = the  most  important  function;  9 = the  least  important 
function). 


Feedback  for  the  clinical  staff. 


Evaluation  of  the  performance  of  individual  staff 
members. 

Evaluation  of  the  quality  of  the  programs  of  the 
mental  health  organization. 

Provide  accountability  to  the  community. 

Provide  accountability  to  the  clientele. 


Increase  the  efficiency  of  the  organization. 


Provide  justification  for  the  funding  of  the  mental 

health  organization. 

Determine  the  allocation  of  resources  among  the 
different  service  programs  within  the  organization. 

Provide  the  means  to  regulate  the  organizational 

members'  behavior. 

Please  rank  the  following  in  what  you  regard  to  be  their  order 
of  importance  in  enabling  an  evaluator  to  establish  an  effective 
evaluation  program.  (1  = the  most  important;  5 = the  least  important). 

The  evaluator's  formal  role  and  powers  in  the 

organization. 

The  evaluator's  personal  friendships  and  relationships 

with  the  clinical  staff. 

The  respect  of  the  clinical  staff  for  the  evaluator's 

technical  skills  and  knowledge. 

The  evaluator's  relationships  with  and  support 

from  the  head  of  the  mental  health  organization. 

The  evaluator's  contributions  to  "non -evaluative'' 

aspects  of  his  job  such  as  data  collection,  grant- 
writing, and  administration. 
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Please  check  the  category  which  most  closely  approximates 
your  position  in  the  organization.  (If  more  than  one  is  appropriate, 
then  check  each --giving  the  percentages  of  time  devoted  to  each 
position.  ) 

Director 


Evaluator 


Clinician 


(Other)  Specify  

How  long  has  an  evaluation  program  been  established  in  your 
organization?  (If  no  evaluation  program  has  been  established,  then 
please  indicate  this.  ) 


How  long  has  the  present  head  of  the  evaluation  staff  been  in 
his  or  her  position? 


Approximately  how  many  full  time  employees  are  on  the  research 
and  evaluation  staff?  (Fractions  may  be  used  to  represent  employees 
who  devote  less  than  full  time  to  research  and  evaluation.  ) 


Approximately  what  percentage  of  the  budget  of  the  total  mental 
health  organization  is  being  devoted  to  evaluation? 


Please  classify  (in  approximate  terms)  the  clientele  of  the 
mental  health  organization: 

Upper  and  upper  middle  class  % 

Middle  and  lower  middle  class  % 

Welfare  and  poor  % 

Approximately  what  percentage  of  the  catchment  area  of  the 
mental  health  organization  can  be  classified  as  urban?  (Urban  is 
defined  here  as  a locality  containing  greater  than  10,000  persons.  ) 
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Indicate  which  (if  any)  of  the  following  programs  are  being 
evaluated  in  your  organization.  Cross  out  all  programs  that  are 
not  currently  being  offered  and  add  (in  the  blank  spaces  provided) 
any  additional  programs  that  are  being  offered  (whether  or  not  they 
are  being  evaluated). 


Is  Being  Evaluated  Is  Not  Being  Evaluated 

Inpatient  

Outpatient  

Emergency  

Partial  

Consultation  

Children  

Rehabilitation  

R and  D 


Approximately  what  is  the  total  number  of  persons  employed 
by  your  mental  health  organization? 
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